2023-04-16 20:53:30,303 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac 2023-04-16 20:53:30,316 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-16 20:53:30,333 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-16 20:53:30,334 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/cluster_f3674968-0d51-0b50-60ff-43cc95ffde93, deleteOnExit=true 2023-04-16 20:53:30,334 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-16 20:53:30,335 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/test.cache.data in system properties and HBase conf 2023-04-16 20:53:30,336 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/hadoop.tmp.dir in system properties and HBase conf 2023-04-16 20:53:30,336 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/hadoop.log.dir in system properties and HBase conf 2023-04-16 20:53:30,337 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-16 20:53:30,337 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-16 20:53:30,337 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-16 20:53:30,453 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-16 20:53:30,867 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-16 20:53:30,871 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-16 20:53:30,872 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-16 20:53:30,872 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-16 20:53:30,872 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-16 20:53:30,873 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-16 20:53:30,873 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-16 20:53:30,873 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-16 20:53:30,874 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-16 20:53:30,874 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-16 20:53:30,874 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/nfs.dump.dir in system properties and HBase conf 2023-04-16 20:53:30,875 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/java.io.tmpdir in system properties and HBase conf 2023-04-16 20:53:30,875 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-16 20:53:30,875 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-16 20:53:30,875 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-16 20:53:31,483 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-16 20:53:31,487 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-16 20:53:31,785 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-16 20:53:31,964 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-16 20:53:31,985 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-16 20:53:32,023 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-16 20:53:32,056 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/java.io.tmpdir/Jetty_localhost_35609_hdfs____.w3ot2x/webapp 2023-04-16 20:53:32,194 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35609 2023-04-16 20:53:32,236 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-16 20:53:32,237 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-16 20:53:32,714 WARN [Listener at localhost/38239] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-16 20:53:32,832 WARN [Listener at localhost/38239] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-16 20:53:32,856 WARN [Listener at localhost/38239] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-16 20:53:32,863 INFO [Listener at localhost/38239] log.Slf4jLog(67): jetty-6.1.26 2023-04-16 20:53:32,867 INFO [Listener at localhost/38239] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/java.io.tmpdir/Jetty_localhost_42259_datanode____vo6xa7/webapp 2023-04-16 20:53:32,964 INFO [Listener at localhost/38239] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42259 2023-04-16 20:53:33,332 WARN [Listener at localhost/45179] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-16 20:53:33,894 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x37b08e9fede21351: Processing first storage report for DS-fbc09a89-c427-423a-8ac0-3b1acd46d1b0 from datanode e6ce2f5f-dc5a-45e0-9c0a-099fa881a9a6 2023-04-16 20:53:33,896 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x37b08e9fede21351: from storage DS-fbc09a89-c427-423a-8ac0-3b1acd46d1b0 node DatanodeRegistration(127.0.0.1:44077, datanodeUuid=e6ce2f5f-dc5a-45e0-9c0a-099fa881a9a6, infoPort=43025, infoSecurePort=0, ipcPort=45179, storageInfo=lv=-57;cid=testClusterID;nsid=527550418;c=1681678411558), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-16 20:53:33,896 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x37b08e9fede21351: Processing first storage report for DS-410977ec-1892-4197-a426-9e8d213f6043 from datanode e6ce2f5f-dc5a-45e0-9c0a-099fa881a9a6 2023-04-16 20:53:33,896 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x37b08e9fede21351: from storage DS-410977ec-1892-4197-a426-9e8d213f6043 node DatanodeRegistration(127.0.0.1:44077, datanodeUuid=e6ce2f5f-dc5a-45e0-9c0a-099fa881a9a6, infoPort=43025, infoSecurePort=0, ipcPort=45179, storageInfo=lv=-57;cid=testClusterID;nsid=527550418;c=1681678411558), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-16 20:53:33,975 DEBUG [Listener at localhost/45179] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac 2023-04-16 20:53:34,032 INFO [Listener at localhost/45179] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/cluster_f3674968-0d51-0b50-60ff-43cc95ffde93/zookeeper_0, clientPort=54578, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/cluster_f3674968-0d51-0b50-60ff-43cc95ffde93/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/cluster_f3674968-0d51-0b50-60ff-43cc95ffde93/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-16 20:53:34,044 INFO [Listener at localhost/45179] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54578 2023-04-16 20:53:34,052 INFO [Listener at localhost/45179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 20:53:34,054 INFO [Listener at localhost/45179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 20:53:34,735 INFO [Listener at localhost/45179] util.FSUtils(471): Created version file at hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad with version=8 2023-04-16 20:53:34,735 INFO [Listener at localhost/45179] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/hbase-staging 2023-04-16 20:53:35,048 INFO [Listener at localhost/45179] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-16 20:53:35,512 INFO [Listener at localhost/45179] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-16 20:53:35,543 INFO [Listener at localhost/45179] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-16 20:53:35,544 INFO [Listener at localhost/45179] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-16 20:53:35,544 INFO [Listener at localhost/45179] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-16 20:53:35,544 INFO [Listener at localhost/45179] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-16 20:53:35,544 INFO [Listener at localhost/45179] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-16 20:53:35,727 INFO [Listener at localhost/45179] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-16 20:53:35,802 DEBUG [Listener at localhost/45179] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-16 20:53:35,895 INFO [Listener at localhost/45179] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46091 2023-04-16 20:53:35,905 INFO [Listener at localhost/45179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 20:53:35,907 INFO [Listener at localhost/45179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 20:53:35,927 INFO [Listener at localhost/45179] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46091 connecting to ZooKeeper ensemble=127.0.0.1:54578 2023-04-16 20:53:35,965 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:460910x0, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-16 20:53:35,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46091-0x1018ec01a880000 connected 2023-04-16 20:53:36,003 DEBUG [Listener at localhost/45179] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-16 20:53:36,004 DEBUG [Listener at localhost/45179] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-16 20:53:36,009 DEBUG [Listener at localhost/45179] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-16 20:53:36,020 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46091 2023-04-16 20:53:36,020 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46091 2023-04-16 20:53:36,021 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46091 2023-04-16 20:53:36,021 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46091 2023-04-16 20:53:36,021 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46091 2023-04-16 20:53:36,027 INFO [Listener at localhost/45179] master.HMaster(444): hbase.rootdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad, hbase.cluster.distributed=false 2023-04-16 20:53:36,093 INFO [Listener at localhost/45179] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-16 20:53:36,094 INFO [Listener at localhost/45179] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-16 20:53:36,094 INFO [Listener at localhost/45179] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-16 20:53:36,094 INFO [Listener at localhost/45179] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-16 20:53:36,094 INFO [Listener at localhost/45179] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-16 20:53:36,094 INFO [Listener at localhost/45179] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-16 20:53:36,099 INFO [Listener at localhost/45179] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-16 20:53:36,102 INFO [Listener at localhost/45179] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41211 2023-04-16 20:53:36,104 INFO [Listener at localhost/45179] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-16 20:53:36,110 DEBUG [Listener at localhost/45179] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-16 20:53:36,111 INFO [Listener at localhost/45179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 20:53:36,113 INFO [Listener at localhost/45179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 20:53:36,115 INFO [Listener at localhost/45179] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41211 connecting to ZooKeeper ensemble=127.0.0.1:54578 2023-04-16 20:53:36,118 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:412110x0, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-16 20:53:36,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41211-0x1018ec01a880001 connected 2023-04-16 20:53:36,119 DEBUG [Listener at localhost/45179] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-16 20:53:36,120 DEBUG [Listener at localhost/45179] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-16 20:53:36,121 DEBUG [Listener at localhost/45179] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-16 20:53:36,121 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41211 2023-04-16 20:53:36,124 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41211 2023-04-16 20:53:36,124 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41211 2023-04-16 20:53:36,128 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41211 2023-04-16 20:53:36,128 DEBUG [Listener at localhost/45179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41211 2023-04-16 20:53:36,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:53:36,142 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-16 20:53:36,144 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:53:36,163 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-16 20:53:36,163 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-16 20:53:36,164 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:53:36,164 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-16 20:53:36,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46091,1681678414874 from backup master directory 2023-04-16 20:53:36,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-16 20:53:36,168 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:53:36,168 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-16 20:53:36,169 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-16 20:53:36,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:53:36,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-16 20:53:36,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-16 20:53:36,259 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/hbase.id with ID: 4a711278-135d-4e64-bb14-db160ef2067d 2023-04-16 20:53:36,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 20:53:36,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:53:36,356 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0020e3a3 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:36,390 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b5c71b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:36,414 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:53:36,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-16 20:53:36,435 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-16 20:53:36,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-16 20:53:36,438 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:53:36,442 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:53:36,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-16 20:53:36,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/data/master/store-tmp 2023-04-16 20:53:36,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:53:36,911 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-16 20:53:36,911 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 20:53:36,911 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 20:53:36,911 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-16 20:53:36,911 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 20:53:36,911 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 20:53:36,911 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-16 20:53:36,913 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/WALs/jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:53:36,937 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46091%2C1681678414874, suffix=, logDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/WALs/jenkins-hbase4.apache.org,46091,1681678414874, archiveDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/oldWALs, maxLogs=10 2023-04-16 20:53:37,008 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44077,DS-fbc09a89-c427-423a-8ac0-3b1acd46d1b0,DISK] 2023-04-16 20:53:37,018 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:53:37,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/WALs/jenkins-hbase4.apache.org,46091,1681678414874/jenkins-hbase4.apache.org%2C46091%2C1681678414874.1681678416948 2023-04-16 20:53:37,090 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44077,DS-fbc09a89-c427-423a-8ac0-3b1acd46d1b0,DISK]] 2023-04-16 20:53:37,091 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:53:37,091 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:53:37,094 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-16 20:53:37,095 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-16 20:53:37,157 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-16 20:53:37,164 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-16 20:53:37,186 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-16 20:53:37,198 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:37,204 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-16 20:53:37,206 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-16 20:53:37,224 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-16 20:53:37,229 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 20:53:37,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64719399, jitterRate=-0.03560580313205719}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-16 20:53:37,231 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-16 20:53:37,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-16 20:53:37,252 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-16 20:53:37,252 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-16 20:53:37,255 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-16 20:53:37,257 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-16 20:53:37,290 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-04-16 20:53:37,290 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-16 20:53:37,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-16 20:53:37,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-16 20:53:37,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-16 20:53:37,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-16 20:53:37,363 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-16 20:53:37,370 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-16 20:53:37,375 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-16 20:53:37,378 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:53:37,381 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-16 20:53:37,382 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-16 20:53:37,394 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-16 20:53:37,398 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-16 20:53:37,398 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-16 20:53:37,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:53:37,399 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,46091,1681678414874, sessionid=0x1018ec01a880000, setting cluster-up flag (Was=false) 2023-04-16 20:53:37,416 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:53:37,425 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-16 20:53:37,426 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:53:37,434 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:53:37,441 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-16 20:53:37,443 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:53:37,446 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.hbase-snapshot/.tmp 2023-04-16 20:53:37,533 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(952): ClusterId : 4a711278-135d-4e64-bb14-db160ef2067d 2023-04-16 20:53:37,542 DEBUG [RS:0;jenkins-hbase4:41211] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-16 20:53:37,549 DEBUG [RS:0;jenkins-hbase4:41211] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-16 20:53:37,549 DEBUG [RS:0;jenkins-hbase4:41211] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-16 20:53:37,552 DEBUG [RS:0;jenkins-hbase4:41211] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-16 20:53:37,553 DEBUG [RS:0;jenkins-hbase4:41211] zookeeper.ReadOnlyZKClient(139): Connect 0x7801a339 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:37,578 DEBUG [RS:0;jenkins-hbase4:41211] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76c64249, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:37,579 DEBUG [RS:0;jenkins-hbase4:41211] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72f9c849, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-16 20:53:37,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-16 20:53:37,612 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:41211 2023-04-16 20:53:37,615 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-16 20:53:37,615 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-16 20:53:37,615 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-16 20:53:37,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-16 20:53:37,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-16 20:53:37,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-16 20:53:37,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,616 INFO [RS:0;jenkins-hbase4:41211] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-16 20:53:37,617 INFO [RS:0;jenkins-hbase4:41211] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-16 20:53:37,617 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1023): About to register with Master. 2023-04-16 20:53:37,620 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46091,1681678414874 with isa=jenkins-hbase4.apache.org/172.31.14.131:41211, startcode=1681678416093 2023-04-16 20:53:37,623 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681678447623 2023-04-16 20:53:37,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-16 20:53:37,628 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-16 20:53:37,629 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-16 20:53:37,635 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-16 20:53:37,636 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-16 20:53:37,642 DEBUG [RS:0;jenkins-hbase4:41211] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-16 20:53:37,646 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-16 20:53:37,646 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-16 20:53:37,647 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-16 20:53:37,647 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-16 20:53:37,648 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,649 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-16 20:53:37,651 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-16 20:53:37,651 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-16 20:53:37,654 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-16 20:53:37,654 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-16 20:53:37,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681678417656,5,FailOnTimeoutGroup] 2023-04-16 20:53:37,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681678417660,5,FailOnTimeoutGroup] 2023-04-16 20:53:37,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,661 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-16 20:53:37,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,679 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-16 20:53:37,680 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-16 20:53:37,680 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad 2023-04-16 20:53:37,696 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:53:37,699 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-16 20:53:37,703 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/info 2023-04-16 20:53:37,704 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-16 20:53:37,705 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:37,706 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-16 20:53:37,710 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45583, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-16 20:53:37,712 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/rep_barrier 2023-04-16 20:53:37,714 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-16 20:53:37,716 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:37,716 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-16 20:53:37,722 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/table 2023-04-16 20:53:37,724 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-16 20:53:37,725 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:37,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:37,727 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740 2023-04-16 20:53:37,728 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740 2023-04-16 20:53:37,733 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:53:37,735 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-16 20:53:37,748 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 20:53:37,750 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69774209, jitterRate=0.03971673548221588}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:53:37,750 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-16 20:53:37,750 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-16 20:53:37,751 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-16 20:53:37,751 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-16 20:53:37,751 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-16 20:53:37,751 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad 2023-04-16 20:53:37,751 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:38239 2023-04-16 20:53:37,751 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-16 20:53:37,751 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-16 20:53:37,753 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-16 20:53:37,753 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-16 20:53:37,759 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-16 20:53:37,760 DEBUG [RS:0;jenkins-hbase4:41211] zookeeper.ZKUtil(162): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:37,760 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-16 20:53:37,760 WARN [RS:0;jenkins-hbase4:41211] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-16 20:53:37,760 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-16 20:53:37,760 INFO [RS:0;jenkins-hbase4:41211] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-16 20:53:37,761 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1947): logDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/WALs/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:37,761 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41211,1681678416093] 2023-04-16 20:53:37,773 DEBUG [RS:0;jenkins-hbase4:41211] zookeeper.ZKUtil(162): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:37,773 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-16 20:53:37,787 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-16 20:53:37,789 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-16 20:53:37,792 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-16 20:53:37,797 INFO [RS:0;jenkins-hbase4:41211] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-16 20:53:37,820 INFO [RS:0;jenkins-hbase4:41211] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-16 20:53:37,823 INFO [RS:0;jenkins-hbase4:41211] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-16 20:53:37,824 INFO [RS:0;jenkins-hbase4:41211] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,829 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-16 20:53:37,837 INFO [RS:0;jenkins-hbase4:41211] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,838 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,838 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,838 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,838 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,839 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,839 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-16 20:53:37,839 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,839 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,839 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,839 DEBUG [RS:0;jenkins-hbase4:41211] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 20:53:37,841 INFO [RS:0;jenkins-hbase4:41211] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,841 INFO [RS:0;jenkins-hbase4:41211] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,841 INFO [RS:0;jenkins-hbase4:41211] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,863 INFO [RS:0;jenkins-hbase4:41211] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-16 20:53:37,866 INFO [RS:0;jenkins-hbase4:41211] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41211,1681678416093-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:37,887 INFO [RS:0;jenkins-hbase4:41211] regionserver.Replication(203): jenkins-hbase4.apache.org,41211,1681678416093 started 2023-04-16 20:53:37,887 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,41211,1681678416093, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41211, sessionid=0x1018ec01a880001 2023-04-16 20:53:37,887 DEBUG [RS:0;jenkins-hbase4:41211] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-16 20:53:37,888 DEBUG [RS:0;jenkins-hbase4:41211] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:37,888 DEBUG [RS:0;jenkins-hbase4:41211] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41211,1681678416093' 2023-04-16 20:53:37,888 DEBUG [RS:0;jenkins-hbase4:41211] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:37,888 DEBUG [RS:0;jenkins-hbase4:41211] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:37,889 DEBUG [RS:0;jenkins-hbase4:41211] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-16 20:53:37,889 DEBUG [RS:0;jenkins-hbase4:41211] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-16 20:53:37,889 DEBUG [RS:0;jenkins-hbase4:41211] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:37,889 DEBUG [RS:0;jenkins-hbase4:41211] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41211,1681678416093' 2023-04-16 20:53:37,889 DEBUG [RS:0;jenkins-hbase4:41211] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-16 20:53:37,890 DEBUG [RS:0;jenkins-hbase4:41211] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-16 20:53:37,890 DEBUG [RS:0;jenkins-hbase4:41211] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-16 20:53:37,890 INFO [RS:0;jenkins-hbase4:41211] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-16 20:53:37,890 INFO [RS:0;jenkins-hbase4:41211] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-16 20:53:37,944 DEBUG [jenkins-hbase4:46091] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-16 20:53:37,947 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41211,1681678416093, state=OPENING 2023-04-16 20:53:37,954 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-16 20:53:37,956 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:53:37,956 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-16 20:53:37,960 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:53:38,001 INFO [RS:0;jenkins-hbase4:41211] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41211%2C1681678416093, suffix=, logDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/WALs/jenkins-hbase4.apache.org,41211,1681678416093, archiveDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/oldWALs, maxLogs=32 2023-04-16 20:53:38,021 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44077,DS-fbc09a89-c427-423a-8ac0-3b1acd46d1b0,DISK] 2023-04-16 20:53:38,024 INFO [RS:0;jenkins-hbase4:41211] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/WALs/jenkins-hbase4.apache.org,41211,1681678416093/jenkins-hbase4.apache.org%2C41211%2C1681678416093.1681678418005 2023-04-16 20:53:38,027 DEBUG [RS:0;jenkins-hbase4:41211] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44077,DS-fbc09a89-c427-423a-8ac0-3b1acd46d1b0,DISK]] 2023-04-16 20:53:38,143 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:38,146 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-16 20:53:38,150 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-16 20:53:38,162 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-16 20:53:38,163 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-16 20:53:38,167 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41211%2C1681678416093.meta, suffix=.meta, logDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/WALs/jenkins-hbase4.apache.org,41211,1681678416093, archiveDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/oldWALs, maxLogs=32 2023-04-16 20:53:38,188 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44077,DS-fbc09a89-c427-423a-8ac0-3b1acd46d1b0,DISK] 2023-04-16 20:53:38,192 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/WALs/jenkins-hbase4.apache.org,41211,1681678416093/jenkins-hbase4.apache.org%2C41211%2C1681678416093.meta.1681678418169.meta 2023-04-16 20:53:38,192 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44077,DS-fbc09a89-c427-423a-8ac0-3b1acd46d1b0,DISK]] 2023-04-16 20:53:38,193 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:53:38,195 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-16 20:53:38,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-16 20:53:38,217 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-16 20:53:38,223 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-16 20:53:38,223 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:53:38,223 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-16 20:53:38,223 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-16 20:53:38,227 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-16 20:53:38,229 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/info 2023-04-16 20:53:38,229 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/info 2023-04-16 20:53:38,229 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-16 20:53:38,230 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:38,230 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-16 20:53:38,232 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/rep_barrier 2023-04-16 20:53:38,232 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/rep_barrier 2023-04-16 20:53:38,232 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-16 20:53:38,233 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:38,234 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-16 20:53:38,235 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/table 2023-04-16 20:53:38,235 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/table 2023-04-16 20:53:38,236 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-16 20:53:38,237 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:38,238 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740 2023-04-16 20:53:38,241 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740 2023-04-16 20:53:38,244 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:53:38,247 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-16 20:53:38,249 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70823609, jitterRate=0.05535401403903961}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:53:38,249 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-16 20:53:38,259 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681678418136 2023-04-16 20:53:38,277 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-16 20:53:38,278 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-16 20:53:38,279 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41211,1681678416093, state=OPEN 2023-04-16 20:53:38,281 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-16 20:53:38,281 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-16 20:53:38,287 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-16 20:53:38,288 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41211,1681678416093 in 321 msec 2023-04-16 20:53:38,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-16 20:53:38,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 516 msec 2023-04-16 20:53:38,299 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 786 msec 2023-04-16 20:53:38,299 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681678418299, completionTime=-1 2023-04-16 20:53:38,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-16 20:53:38,300 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-16 20:53:38,366 DEBUG [hconnection-0x1e748436-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:38,371 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43132, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:38,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-16 20:53:38,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681678478387 2023-04-16 20:53:38,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681678538387 2023-04-16 20:53:38,388 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 87 msec 2023-04-16 20:53:38,414 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46091,1681678414874-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:38,414 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46091,1681678414874-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:38,414 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46091,1681678414874-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:38,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46091, period=300000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:38,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-16 20:53:38,423 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-16 20:53:38,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-16 20:53:38,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-16 20:53:38,441 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-16 20:53:38,443 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 20:53:38,446 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 20:53:38,468 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:53:38,470 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6 empty. 2023-04-16 20:53:38,471 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:53:38,471 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-16 20:53:38,895 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-16 20:53:38,897 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 707804278a44b6c4590cd788e4bf28c6, NAME => 'hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp 2023-04-16 20:53:38,909 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:53:38,909 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 707804278a44b6c4590cd788e4bf28c6, disabling compactions & flushes 2023-04-16 20:53:38,909 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:53:38,909 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:53:38,909 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. after waiting 0 ms 2023-04-16 20:53:38,909 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:53:38,909 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:53:38,909 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 707804278a44b6c4590cd788e4bf28c6: 2023-04-16 20:53:38,914 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 20:53:38,929 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681678418917"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678418917"}]},"ts":"1681678418917"} 2023-04-16 20:53:38,958 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 20:53:38,960 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 20:53:38,964 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678418960"}]},"ts":"1681678418960"} 2023-04-16 20:53:38,969 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-16 20:53:38,980 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=707804278a44b6c4590cd788e4bf28c6, ASSIGN}] 2023-04-16 20:53:38,984 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=707804278a44b6c4590cd788e4bf28c6, ASSIGN 2023-04-16 20:53:38,985 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=707804278a44b6c4590cd788e4bf28c6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=false 2023-04-16 20:53:39,136 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=707804278a44b6c4590cd788e4bf28c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:39,137 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681678419136"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678419136"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678419136"}]},"ts":"1681678419136"} 2023-04-16 20:53:39,143 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 707804278a44b6c4590cd788e4bf28c6, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:53:39,303 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:53:39,304 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 707804278a44b6c4590cd788e4bf28c6, NAME => 'hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:53:39,305 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:53:39,305 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:53:39,305 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:53:39,306 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:53:39,308 INFO [StoreOpener-707804278a44b6c4590cd788e4bf28c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:53:39,311 DEBUG [StoreOpener-707804278a44b6c4590cd788e4bf28c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6/info 2023-04-16 20:53:39,311 DEBUG [StoreOpener-707804278a44b6c4590cd788e4bf28c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6/info 2023-04-16 20:53:39,311 INFO [StoreOpener-707804278a44b6c4590cd788e4bf28c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 707804278a44b6c4590cd788e4bf28c6 columnFamilyName info 2023-04-16 20:53:39,312 INFO [StoreOpener-707804278a44b6c4590cd788e4bf28c6-1] regionserver.HStore(310): Store=707804278a44b6c4590cd788e4bf28c6/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:39,314 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:53:39,315 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:53:39,320 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:53:39,324 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 20:53:39,325 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 707804278a44b6c4590cd788e4bf28c6; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60493256, jitterRate=-0.0985802412033081}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-16 20:53:39,325 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 707804278a44b6c4590cd788e4bf28c6: 2023-04-16 20:53:39,327 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6., pid=6, masterSystemTime=1681678419297 2023-04-16 20:53:39,331 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:53:39,331 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:53:39,333 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=707804278a44b6c4590cd788e4bf28c6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:39,334 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681678419332"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678419332"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678419332"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678419332"}]},"ts":"1681678419332"} 2023-04-16 20:53:39,341 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-16 20:53:39,341 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 707804278a44b6c4590cd788e4bf28c6, server=jenkins-hbase4.apache.org,41211,1681678416093 in 194 msec 2023-04-16 20:53:39,345 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-16 20:53:39,345 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=707804278a44b6c4590cd788e4bf28c6, ASSIGN in 361 msec 2023-04-16 20:53:39,346 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 20:53:39,347 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678419347"}]},"ts":"1681678419347"} 2023-04-16 20:53:39,350 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-16 20:53:39,354 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 20:53:39,357 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 922 msec 2023-04-16 20:53:39,444 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-16 20:53:39,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-16 20:53:39,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:53:39,485 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-16 20:53:39,501 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-16 20:53:39,508 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-04-16 20:53:39,519 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-16 20:53:39,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-16 20:53:39,534 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-04-16 20:53:39,545 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-16 20:53:39,547 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-16 20:53:39,548 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.378sec 2023-04-16 20:53:39,549 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-16 20:53:39,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-16 20:53:39,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-16 20:53:39,552 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46091,1681678414874-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-16 20:53:39,553 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46091,1681678414874-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-16 20:53:39,563 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-16 20:53:39,638 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:39,642 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:39,654 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:39,665 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43148, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:39,674 INFO [Listener at localhost/45179] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:53:39,693 INFO [Listener at localhost/45179] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=169, AvailableMemoryMB=1407 2023-04-16 20:53:39,709 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 20:53:39,712 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48396, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 20:53:39,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 20:53:39,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:53:39,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 20:53:39,731 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 20:53:39,733 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 20:53:39,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-16 20:53:39,736 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:39,737 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c empty. 2023-04-16 20:53:39,739 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:39,739 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:53:39,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-16 20:53:39,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-16 20:53:40,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-16 20:53:40,156 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 20:53:40,159 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => bfe0529baac322a72dbf6ef88dd5ff1c, NAME => 'TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp 2023-04-16 20:53:40,170 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:53:40,171 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing bfe0529baac322a72dbf6ef88dd5ff1c, disabling compactions & flushes 2023-04-16 20:53:40,171 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:40,171 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:40,171 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. after waiting 0 ms 2023-04-16 20:53:40,171 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:40,171 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:40,171 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:40,175 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 20:53:40,176 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678420176"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678420176"}]},"ts":"1681678420176"} 2023-04-16 20:53:40,179 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 20:53:40,180 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 20:53:40,180 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678420180"}]},"ts":"1681678420180"} 2023-04-16 20:53:40,182 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 20:53:40,186 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfe0529baac322a72dbf6ef88dd5ff1c, ASSIGN}] 2023-04-16 20:53:40,188 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfe0529baac322a72dbf6ef88dd5ff1c, ASSIGN 2023-04-16 20:53:40,189 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfe0529baac322a72dbf6ef88dd5ff1c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=false 2023-04-16 20:53:40,340 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=bfe0529baac322a72dbf6ef88dd5ff1c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:40,341 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678420340"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678420340"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678420340"}]},"ts":"1681678420340"} 2023-04-16 20:53:40,344 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:53:40,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-16 20:53:40,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:40,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bfe0529baac322a72dbf6ef88dd5ff1c, NAME => 'TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:53:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:53:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:40,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:40,507 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:40,509 DEBUG [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A 2023-04-16 20:53:40,510 DEBUG [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A 2023-04-16 20:53:40,511 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:53:40,512 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bfe0529baac322a72dbf6ef88dd5ff1c columnFamilyName A 2023-04-16 20:53:40,513 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] regionserver.HStore(310): Store=bfe0529baac322a72dbf6ef88dd5ff1c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:40,513 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:40,515 DEBUG [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B 2023-04-16 20:53:40,515 DEBUG [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B 2023-04-16 20:53:40,515 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:53:40,516 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bfe0529baac322a72dbf6ef88dd5ff1c columnFamilyName B 2023-04-16 20:53:40,517 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] regionserver.HStore(310): Store=bfe0529baac322a72dbf6ef88dd5ff1c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:40,517 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:40,519 DEBUG [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C 2023-04-16 20:53:40,519 DEBUG [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C 2023-04-16 20:53:40,519 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:53:40,520 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bfe0529baac322a72dbf6ef88dd5ff1c columnFamilyName C 2023-04-16 20:53:40,521 INFO [StoreOpener-bfe0529baac322a72dbf6ef88dd5ff1c-1] regionserver.HStore(310): Store=bfe0529baac322a72dbf6ef88dd5ff1c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:53:40,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:40,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:40,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:40,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:53:40,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:40,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 20:53:40,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bfe0529baac322a72dbf6ef88dd5ff1c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72602823, jitterRate=0.08186636865139008}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:53:40,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:40,538 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., pid=11, masterSystemTime=1681678420497 2023-04-16 20:53:40,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:40,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:40,544 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=bfe0529baac322a72dbf6ef88dd5ff1c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:40,544 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678420543"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678420543"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678420543"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678420543"}]},"ts":"1681678420543"} 2023-04-16 20:53:40,553 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-16 20:53:40,553 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 in 205 msec 2023-04-16 20:53:40,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-16 20:53:40,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfe0529baac322a72dbf6ef88dd5ff1c, ASSIGN in 367 msec 2023-04-16 20:53:40,558 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 20:53:40,559 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678420558"}]},"ts":"1681678420558"} 2023-04-16 20:53:40,561 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 20:53:40,564 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 20:53:40,567 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 838 msec 2023-04-16 20:53:40,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-16 20:53:40,857 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-16 20:53:40,862 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x2678d495 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,875 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8c3f619, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,878 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:40,881 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:40,884 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 20:53:40,887 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 20:53:40,895 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x1b886c8f to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,899 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748f869b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,899 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,904 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e91d243, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,904 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x77648431 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,909 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,909 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,913 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46192d17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,913 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x09a575b8 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,917 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c57e1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,918 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x1818ad0a to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,923 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d71120, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,923 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,955 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,956 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x6129209e to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,959 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d24bbb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,960 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x4b02eeb3 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:53:40,963 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63871a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:53:40,968 DEBUG [hconnection-0x685c3eb3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:40,968 DEBUG [hconnection-0x1865dd73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:40,977 DEBUG [hconnection-0x2b90131a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:40,978 DEBUG [hconnection-0x4896b578-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:40,979 DEBUG [hconnection-0x2be07eef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:40,979 DEBUG [hconnection-0x79e1e112-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:40,981 DEBUG [hconnection-0x139a00b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:40,981 DEBUG [hconnection-0x635c9341-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:40,986 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:40,994 DEBUG [hconnection-0x1b518f7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:53:41,002 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43174, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:41,002 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:41,003 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:41,003 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:41,004 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:41,007 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43186, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:41,008 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:41,013 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43230, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:53:41,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:41,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:41,077 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:41,078 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:41,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:41,081 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:41,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:41,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:41,091 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:41,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:41,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:41,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:41,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:41,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:41,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:41,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:41,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:41,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:41,128 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:41,131 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:41,133 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-16 20:53:41,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:41,138 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:41,172 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:41,173 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:41,173 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:41,173 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:41,173 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:41,173 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:41,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678481270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678421239, max timestamp=1681678421273Num uniques 859; 2023-04-16 20:53:41,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:41,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678421239, max timestamp=1681678421273Num uniques 859; 2023-04-16 20:53:41,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 20:53:41,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678421239, max timestamp=1681678421271Num uniques 859; 2023-04-16 20:53:41,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 20:53:41,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678481354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,362 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/771aea0c8f1f435f9add8dcb37b02c11 2023-04-16 20:53:41,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678481356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678481358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678481358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678481421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678481466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678481468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678481468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678481469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,550 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/653206bc1a1a48c29dfb02a6f3fee84d 2023-04-16 20:53:41,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678481630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678481675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678481676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678481678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678481679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678481941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678481979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678481981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678481986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:41,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:41,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678481986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,048 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/82a5f07124c54708973e89534e736a15 2023-04-16 20:53:42,062 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/771aea0c8f1f435f9add8dcb37b02c11 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/771aea0c8f1f435f9add8dcb37b02c11 2023-04-16 20:53:42,075 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/771aea0c8f1f435f9add8dcb37b02c11, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 20:53:42,079 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/653206bc1a1a48c29dfb02a6f3fee84d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/653206bc1a1a48c29dfb02a6f3fee84d 2023-04-16 20:53:42,109 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/653206bc1a1a48c29dfb02a6f3fee84d, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 20:53:42,111 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/82a5f07124c54708973e89534e736a15 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/82a5f07124c54708973e89534e736a15 2023-04-16 20:53:42,123 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/82a5f07124c54708973e89534e736a15, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 20:53:42,126 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.68 KB/129720 for bfe0529baac322a72dbf6ef88dd5ff1c in 993ms, sequenceid=12, compaction requested=false 2023-04-16 20:53:42,128 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-16 20:53:42,130 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:42,130 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:42,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:42,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:42,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:42,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:42,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:42,134 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:42,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:42,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:42,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:42,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ffc081b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:42,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:42,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:42,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:42,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:42,144 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:42,144 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:42,144 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:42,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:42,153 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:42,153 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,153 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:42,153 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:42,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:42,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:42,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:42,160 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:42,160 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:42,160 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:42,160 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:42,160 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:42,166 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:42,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:42,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:42,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:42,170 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:42,172 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:42,176 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,192 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:42,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:42,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:42,192 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,192 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:42,192 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:42,192 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,192 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:42,192 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:42,193 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:42,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:42,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:42,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:42,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:42,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:42,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:42,199 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:42,199 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:42,299 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:42,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:42,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:42,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:42,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:42,340 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:42,340 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:42,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:42,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:42,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:42,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:42,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:42,350 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:42,351 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,351 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:42,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:42,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:42,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:42,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:42,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:42,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:42,363 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:42,363 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:42,369 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:42,369 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:42,369 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:42,369 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:42,370 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=126.68 KB heapSize=363.23 KB 2023-04-16 20:53:42,370 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:42,371 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:42,371 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:42,371 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:42,371 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:42,371 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:42,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:42,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:42,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678482506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678482508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678482511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678482512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678482515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678482622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678482623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678482623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678482629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678482629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678482829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678482830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678482831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678482833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:42,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678482835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:42,900 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/6a7fa4c7900b488eb5dcc415309fbb39 2023-04-16 20:53:42,972 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/1baa6fa1b65b41f7a25211563987dd3d 2023-04-16 20:53:43,041 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/d0e7d5bc6dd64a139cca0dcda20c669e 2023-04-16 20:53:43,052 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/6a7fa4c7900b488eb5dcc415309fbb39 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6a7fa4c7900b488eb5dcc415309fbb39 2023-04-16 20:53:43,061 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6a7fa4c7900b488eb5dcc415309fbb39, entries=150, sequenceid=48, filesize=11.6 K 2023-04-16 20:53:43,064 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/1baa6fa1b65b41f7a25211563987dd3d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1baa6fa1b65b41f7a25211563987dd3d 2023-04-16 20:53:43,073 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1baa6fa1b65b41f7a25211563987dd3d, entries=150, sequenceid=48, filesize=11.6 K 2023-04-16 20:53:43,076 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/d0e7d5bc6dd64a139cca0dcda20c669e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d0e7d5bc6dd64a139cca0dcda20c669e 2023-04-16 20:53:43,092 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d0e7d5bc6dd64a139cca0dcda20c669e, entries=150, sequenceid=48, filesize=11.6 K 2023-04-16 20:53:43,094 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.68 KB/129720, heapSize ~363.19 KB/371904, currentSize=60.38 KB/61830 for bfe0529baac322a72dbf6ef88dd5ff1c in 724ms, sequenceid=48, compaction requested=false 2023-04-16 20:53:43,094 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:43,094 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:43,097 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:43,097 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,097 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:43,097 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:43,102 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:43,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31dc2125[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:43,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:43,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:43,113 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:43,113 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:43,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:43,117 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:43,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:43,118 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:43,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:43,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:43,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:43,118 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:43,123 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,123 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,124 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,129 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,136 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:43,136 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:43,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:43,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:43,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:43,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:43,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,141 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:43,141 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:43,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:43,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:43,145 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:53:43,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:43,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:43,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:43,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:43,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:43,149 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:43,149 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:43,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678483177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678483180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678483181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678483182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678483194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,241 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:43,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:43,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:43,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:43,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:43,269 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:43,269 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:43,273 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:43,273 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:43,280 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,280 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:43,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:43,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,283 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,283 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:43,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:43,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:43,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:43,296 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:43,299 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:43,299 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:43,300 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,300 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:43,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678483299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,300 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:43,301 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,305 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:43,305 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:43,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,305 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,305 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:43,305 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:43,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678483300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,309 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,310 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,310 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:43,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678483307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678483307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678483313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:43,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@639b1c78[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:43,322 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:43,322 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,323 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:43,324 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,324 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:43,329 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:43,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:43,340 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:43,340 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:43,340 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:43,340 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:43,340 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:43,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:43,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,350 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:43,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:43,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:43,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:43,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,360 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:43,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:43,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:43,362 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:43,362 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:43,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:43,462 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:43,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:43,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:43,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:43,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:43,480 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:43,480 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:43,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:43,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:43,482 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,482 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,484 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:43,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:43,484 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:43,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:43,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:43,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:43,489 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:43,489 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:43,489 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:43,489 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,490 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:43,490 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:43,490 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,495 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:43,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44ed34da[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:43,499 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:43,499 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,502 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:43,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,503 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:43,503 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:43,503 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:43,509 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,509 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,509 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:43,509 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678483510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678483510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:43,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:43,515 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:43,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:43,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:43,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:43,515 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:43,518 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,518 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:43,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:43,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678483519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678483519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678483519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,525 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:43,540 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,541 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,541 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,541 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,541 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,541 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,541 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:43,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:43,542 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:43,542 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:43,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:43,542 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:43,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:43,542 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:43,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:43,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/5814f3b24e5b44d49e7430751ee3deda 2023-04-16 20:53:43,642 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:43,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:43,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:43,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:43,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:43,667 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:43,667 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:43,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:43,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:43,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:43,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:43,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:43,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:43,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:43,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:43,686 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:43,686 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:43,687 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:43,688 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:43,688 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:43,688 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,688 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:43,688 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:43,689 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:43,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:43,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:43,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:43,694 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,694 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:43,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b9e508c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:43,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:43,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,703 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,703 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:43,703 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:43,703 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:43,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:43,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:43,706 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:43,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:43,711 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:43,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:43,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:43,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:43,711 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:43,721 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,721 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,721 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:43,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:43,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,736 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:43,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/9b09ed514ecc49cdbf3c02c3180bdea2 2023-04-16 20:53:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,746 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:43,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:43,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:43,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:43,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:43,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:43,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:43,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:43,750 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:43,751 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:43,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:43,785 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 20:53:43,793 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/6f385e17007844618c3fea5c3983908f 2023-04-16 20:53:43,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/5814f3b24e5b44d49e7430751ee3deda as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5814f3b24e5b44d49e7430751ee3deda 2023-04-16 20:53:43,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678483818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,823 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5814f3b24e5b44d49e7430751ee3deda, entries=150, sequenceid=62, filesize=11.6 K 2023-04-16 20:53:43,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678483818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/9b09ed514ecc49cdbf3c02c3180bdea2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9b09ed514ecc49cdbf3c02c3180bdea2 2023-04-16 20:53:43,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9b09ed514ecc49cdbf3c02c3180bdea2, entries=150, sequenceid=62, filesize=11.6 K 2023-04-16 20:53:43,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678483828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:43,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678483828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/6f385e17007844618c3fea5c3983908f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f385e17007844618c3fea5c3983908f 2023-04-16 20:53:43,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678483827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,852 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:43,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:43,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:43,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f385e17007844618c3fea5c3983908f, entries=150, sequenceid=62, filesize=11.6 K 2023-04-16 20:53:43,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for bfe0529baac322a72dbf6ef88dd5ff1c in 736ms, sequenceid=62, compaction requested=true 2023-04-16 20:53:43,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:43,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:43,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:43,886 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:43,887 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:43,902 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:43,902 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:43,903 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,903 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:43,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:43,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:43,904 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:43,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:43,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:43,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:43,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:43,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:43,909 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:43,912 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:43,912 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,912 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/653206bc1a1a48c29dfb02a6f3fee84d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1baa6fa1b65b41f7a25211563987dd3d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9b09ed514ecc49cdbf3c02c3180bdea2] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=34.8 K 2023-04-16 20:53:43,914 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 653206bc1a1a48c29dfb02a6f3fee84d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681678421096 2023-04-16 20:53:43,915 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1baa6fa1b65b41f7a25211563987dd3d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681678421337 2023-04-16 20:53:43,916 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9b09ed514ecc49cdbf3c02c3180bdea2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678422507 2023-04-16 20:53:43,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:43,924 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:43,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:43,926 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:43,926 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:43,934 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:43,935 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,935 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:43,939 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-16 20:53:43,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:43,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:43,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:43,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:43,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:43,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:43,953 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:43,954 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:43,954 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:43,954 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/771aea0c8f1f435f9add8dcb37b02c11, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6a7fa4c7900b488eb5dcc415309fbb39, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5814f3b24e5b44d49e7430751ee3deda] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=34.8 K 2023-04-16 20:53:43,968 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 771aea0c8f1f435f9add8dcb37b02c11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681678421096 2023-04-16 20:53:43,969 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6a7fa4c7900b488eb5dcc415309fbb39, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681678421337 2023-04-16 20:53:43,970 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 5814f3b24e5b44d49e7430751ee3deda, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678422507 2023-04-16 20:53:43,977 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-16 20:53:44,011 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-16 20:53:44,021 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#10 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:44,023 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#11 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:44,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:44,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:44,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678484355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678484358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678484361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678484361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678484362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678484464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678484464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678484471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678484474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678484475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,523 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/5547f4488857420a972492f8569e8b33 2023-04-16 20:53:44,584 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/1892409c287148feaad8141461d04a38 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1892409c287148feaad8141461d04a38 2023-04-16 20:53:44,590 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/4cb5ad0846924860bb505e90d509320b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/4cb5ad0846924860bb505e90d509320b 2023-04-16 20:53:44,627 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into 1892409c287148feaad8141461d04a38(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:44,628 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:44,628 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=13, startTime=1681678423883; duration=0sec 2023-04-16 20:53:44,629 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:44,629 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:44,637 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:44,638 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:44,638 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:44,638 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/82a5f07124c54708973e89534e736a15, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d0e7d5bc6dd64a139cca0dcda20c669e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f385e17007844618c3fea5c3983908f] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=34.8 K 2023-04-16 20:53:44,639 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 82a5f07124c54708973e89534e736a15, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681678421096 2023-04-16 20:53:44,639 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into 4cb5ad0846924860bb505e90d509320b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:44,641 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:44,641 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=13, startTime=1681678423904; duration=0sec 2023-04-16 20:53:44,641 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:44,642 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d0e7d5bc6dd64a139cca0dcda20c669e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681678421337 2023-04-16 20:53:44,642 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6f385e17007844618c3fea5c3983908f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678422507 2023-04-16 20:53:44,664 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#13 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:44,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678484673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678484674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678484681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678484682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678484682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,723 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/68dab8a75b3b40bbba9710adfbdb948b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/68dab8a75b3b40bbba9710adfbdb948b 2023-04-16 20:53:44,737 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into 68dab8a75b3b40bbba9710adfbdb948b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:44,737 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:44,737 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=13, startTime=1681678423904; duration=0sec 2023-04-16 20:53:44,737 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:44,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678484979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678484982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678484986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:44,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:44,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678484988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678484988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,039 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/f6ee1c9feb3d4979997c1d3659bf6280 2023-04-16 20:53:45,044 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-16 20:53:45,045 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-16 20:53:45,108 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/3f880bd189fa4949af26caac844296e4 2023-04-16 20:53:45,122 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/5547f4488857420a972492f8569e8b33 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5547f4488857420a972492f8569e8b33 2023-04-16 20:53:45,136 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5547f4488857420a972492f8569e8b33, entries=150, sequenceid=80, filesize=11.6 K 2023-04-16 20:53:45,139 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/f6ee1c9feb3d4979997c1d3659bf6280 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6ee1c9feb3d4979997c1d3659bf6280 2023-04-16 20:53:45,165 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6ee1c9feb3d4979997c1d3659bf6280, entries=150, sequenceid=80, filesize=11.6 K 2023-04-16 20:53:45,168 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/3f880bd189fa4949af26caac844296e4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3f880bd189fa4949af26caac844296e4 2023-04-16 20:53:45,189 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3f880bd189fa4949af26caac844296e4, entries=150, sequenceid=80, filesize=11.6 K 2023-04-16 20:53:45,195 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for bfe0529baac322a72dbf6ef88dd5ff1c in 1256ms, sequenceid=80, compaction requested=false 2023-04-16 20:53:45,195 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:45,196 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:45,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:45,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:45,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,197 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:45,197 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:45,198 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,199 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:45,200 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:45,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:45,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1b9dd3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:45,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:45,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,204 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,205 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:45,205 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:45,205 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:45,208 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:45,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:45,211 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:45,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:45,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:45,219 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:45,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:45,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:45,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:45,219 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:45,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:45,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:45,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:45,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:45,229 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,234 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:45,234 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:45,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:45,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:45,234 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:45,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:45,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:45,234 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,237 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,237 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,237 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:45,237 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:45,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:45,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:45,241 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,242 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,242 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,242 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,337 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:45,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:45,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:45,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:45,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:45,364 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:45,364 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:45,373 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:45,373 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:45,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:45,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:45,384 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:45,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:45,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:45,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:45,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:45,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:45,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:45,390 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:45,390 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:45,390 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-16 20:53:45,394 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:45,394 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:45,394 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:45,394 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:45,394 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:45,394 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:45,491 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/c10bcd44c0b7456aa1617f5ac675ddb3 2023-04-16 20:53:45,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:45,499 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:45,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678485525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678485526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678485527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678485528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678485529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,559 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/34d8daa1e8124b289628dc4c699931f8 2023-04-16 20:53:45,594 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/6e3ca267fe05482c84c67369e4d738a0 2023-04-16 20:53:45,612 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/c10bcd44c0b7456aa1617f5ac675ddb3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c10bcd44c0b7456aa1617f5ac675ddb3 2023-04-16 20:53:45,622 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c10bcd44c0b7456aa1617f5ac675ddb3, entries=150, sequenceid=98, filesize=11.6 K 2023-04-16 20:53:45,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/34d8daa1e8124b289628dc4c699931f8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/34d8daa1e8124b289628dc4c699931f8 2023-04-16 20:53:45,634 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/34d8daa1e8124b289628dc4c699931f8, entries=150, sequenceid=98, filesize=11.6 K 2023-04-16 20:53:45,637 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/6e3ca267fe05482c84c67369e4d738a0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6e3ca267fe05482c84c67369e4d738a0 2023-04-16 20:53:45,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678485633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678485633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678485633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678485633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678485635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,650 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6e3ca267fe05482c84c67369e4d738a0, entries=150, sequenceid=98, filesize=11.6 K 2023-04-16 20:53:45,660 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for bfe0529baac322a72dbf6ef88dd5ff1c in 270ms, sequenceid=98, compaction requested=true 2023-04-16 20:53:45,660 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:45,660 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:45,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:45,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:45,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:45,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:45,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:45,664 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,664 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:45,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:45,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d82214[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:45,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:45,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,670 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:45,670 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:45,670 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:45,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:45,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:45,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:45,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:45,675 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:45,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:45,681 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:45,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:45,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:45,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:45,681 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:45,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:45,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:45,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:45,685 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:45,687 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:45,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:45,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:45,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:45,692 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:45,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:45,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:45,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:45,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,699 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:45,699 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:45,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:45,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:45,799 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:45,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:45,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:45,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:45,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:45,806 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:45,806 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:45,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:45,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:45,808 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,808 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:45,812 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:45,813 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,813 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:45,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:45,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:45,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:45,817 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:45,817 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:45,821 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:45,824 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:45,824 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:45,824 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-16 20:53:45,824 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:45,825 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:45,825 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:45,825 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:45,825 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:45,825 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:45,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:45,842 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:45,871 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/c137c5010e1b46548e3306d5017d3fc7 2023-04-16 20:53:45,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678485858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678485859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678485865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678485867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678485873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,902 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/feed592d0d064a3fa2afa3914bced109 2023-04-16 20:53:45,947 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/6f4fef4a71814b4ba08cf9c9ee16c620 2023-04-16 20:53:45,960 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/c137c5010e1b46548e3306d5017d3fc7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c137c5010e1b46548e3306d5017d3fc7 2023-04-16 20:53:45,971 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c137c5010e1b46548e3306d5017d3fc7, entries=150, sequenceid=115, filesize=11.6 K 2023-04-16 20:53:45,974 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/feed592d0d064a3fa2afa3914bced109 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/feed592d0d064a3fa2afa3914bced109 2023-04-16 20:53:45,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678485976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678485976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678485977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678485977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:45,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678485987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:45,997 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/feed592d0d064a3fa2afa3914bced109, entries=150, sequenceid=115, filesize=11.6 K 2023-04-16 20:53:46,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/6f4fef4a71814b4ba08cf9c9ee16c620 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f4fef4a71814b4ba08cf9c9ee16c620 2023-04-16 20:53:46,014 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f4fef4a71814b4ba08cf9c9ee16c620, entries=150, sequenceid=115, filesize=11.6 K 2023-04-16 20:53:46,017 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for bfe0529baac322a72dbf6ef88dd5ff1c in 193ms, sequenceid=115, compaction requested=true 2023-04-16 20:53:46,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:46,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,017 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:46,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:46,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:46,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:46,026 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,026 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:46,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dbe8131[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:46,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:46,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:46,034 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:46,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:46,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:46,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:46,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:46,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:46,056 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:46,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:46,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:46,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:46,056 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:46,060 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,065 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,069 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,082 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,082 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,082 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,083 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,083 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,083 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:46,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:46,083 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,087 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,083 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,087 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:46,087 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:46,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,187 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:46,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:46,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-16 20:53:46,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:46,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:46,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:46,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:46,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:46,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:46,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:46,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:46,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:46,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:46,206 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:46,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:46,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:46,226 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,226 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,227 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,227 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678486214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,228 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678486216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,228 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:46,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678486219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678486228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678486228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:46,245 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:46,245 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:46,247 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,247 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:46,247 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:46,247 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:46,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:46,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:46,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:46,253 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:46,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34dda23a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:46,257 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:46,257 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:46,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:46,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:46,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:46,260 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:46,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:46,272 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:46,273 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:46,273 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:46,273 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:46,273 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:46,278 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,278 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,279 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,280 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,281 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,297 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,297 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,297 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,297 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,297 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,298 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,298 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,298 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,298 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,298 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:46,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:46,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,301 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:46,301 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:46,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/bc4a5df7870443d3b2b3bfa74ff7773d 2023-04-16 20:53:46,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678486335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678486334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678486335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678486336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678486336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/be62e4ebb62340dd8c367c8e3162791f 2023-04-16 20:53:46,381 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/e497b7d46e2a4e61bc24c1ee5e33206b 2023-04-16 20:53:46,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/bc4a5df7870443d3b2b3bfa74ff7773d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/bc4a5df7870443d3b2b3bfa74ff7773d 2023-04-16 20:53:46,401 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:46,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:46,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:46,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/bc4a5df7870443d3b2b3bfa74ff7773d, entries=250, sequenceid=132, filesize=16.3 K 2023-04-16 20:53:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:46,408 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/be62e4ebb62340dd8c367c8e3162791f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/be62e4ebb62340dd8c367c8e3162791f 2023-04-16 20:53:46,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:46,409 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:46,409 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:46,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:46,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,411 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,411 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:46,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:46,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:46,415 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:46,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,418 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:46,420 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,420 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:46,420 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:46,420 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:46,425 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,425 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,425 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:46,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@332f5477[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:46,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:46,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,429 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:46,430 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:46,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:46,433 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:46,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:46,442 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:46,442 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:46,442 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:46,442 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:46,442 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:46,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/be62e4ebb62340dd8c367c8e3162791f, entries=150, sequenceid=132, filesize=11.7 K 2023-04-16 20:53:46,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/e497b7d46e2a4e61bc24c1ee5e33206b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/e497b7d46e2a4e61bc24c1ee5e33206b 2023-04-16 20:53:46,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,454 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,454 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,455 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,455 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,455 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,455 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:46,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,455 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,456 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,456 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,456 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,456 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:46,456 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:46,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:46,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/e497b7d46e2a4e61bc24c1ee5e33206b, entries=150, sequenceid=132, filesize=11.7 K 2023-04-16 20:53:46,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for bfe0529baac322a72dbf6ef88dd5ff1c in 274ms, sequenceid=132, compaction requested=true 2023-04-16 20:53:46,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:46,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:46,463 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:53:46,463 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:53:46,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:46,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:46,466 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64363 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:53:46,466 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:46,466 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,467 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1892409c287148feaad8141461d04a38, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5547f4488857420a972492f8569e8b33, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c10bcd44c0b7456aa1617f5ac675ddb3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c137c5010e1b46548e3306d5017d3fc7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/bc4a5df7870443d3b2b3bfa74ff7773d] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=62.9 K 2023-04-16 20:53:46,467 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59633 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:53:46,467 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1892409c287148feaad8141461d04a38, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678422507 2023-04-16 20:53:46,468 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:46,469 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,469 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/4cb5ad0846924860bb505e90d509320b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6ee1c9feb3d4979997c1d3659bf6280, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/34d8daa1e8124b289628dc4c699931f8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/feed592d0d064a3fa2afa3914bced109, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/be62e4ebb62340dd8c367c8e3162791f] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=58.2 K 2023-04-16 20:53:46,469 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 5547f4488857420a972492f8569e8b33, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681678423177 2023-04-16 20:53:46,470 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 4cb5ad0846924860bb505e90d509320b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678422507 2023-04-16 20:53:46,470 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c10bcd44c0b7456aa1617f5ac675ddb3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681678424334 2023-04-16 20:53:46,472 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting f6ee1c9feb3d4979997c1d3659bf6280, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681678423177 2023-04-16 20:53:46,472 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c137c5010e1b46548e3306d5017d3fc7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681678425513 2023-04-16 20:53:46,472 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 34d8daa1e8124b289628dc4c699931f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681678424334 2023-04-16 20:53:46,473 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting bc4a5df7870443d3b2b3bfa74ff7773d, keycount=250, bloomtype=ROW, size=16.3 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681678425853 2023-04-16 20:53:46,473 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting feed592d0d064a3fa2afa3914bced109, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681678425513 2023-04-16 20:53:46,474 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting be62e4ebb62340dd8c367c8e3162791f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681678425853 2023-04-16 20:53:46,497 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#24 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:46,510 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#25 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:46,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-16 20:53:46,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:46,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:46,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:46,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:46,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:46,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:46,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:46,556 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:46,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:46,561 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/137d9bf6306d4179a7991bb6ce3acbfb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/137d9bf6306d4179a7991bb6ce3acbfb 2023-04-16 20:53:46,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:46,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:46,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:46,567 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:46,567 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,570 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:46,573 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:46,574 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,574 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,575 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/be19d543fc5f4077b89c906df5b71870 2023-04-16 20:53:46,576 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into 137d9bf6306d4179a7991bb6ce3acbfb(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:46,576 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,576 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:46,576 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=11, startTime=1681678426463; duration=0sec 2023-04-16 20:53:46,576 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,576 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:46,576 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:53:46,577 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,577 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:46,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:46,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:46,578 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:46,578 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:46,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:46,580 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,580 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:46,580 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:46,580 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,581 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59633 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:53:46,581 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:46,581 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,582 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/68dab8a75b3b40bbba9710adfbdb948b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3f880bd189fa4949af26caac844296e4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6e3ca267fe05482c84c67369e4d738a0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f4fef4a71814b4ba08cf9c9ee16c620, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/e497b7d46e2a4e61bc24c1ee5e33206b] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=58.2 K 2023-04-16 20:53:46,582 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 68dab8a75b3b40bbba9710adfbdb948b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678422507 2023-04-16 20:53:46,583 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3f880bd189fa4949af26caac844296e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681678423177 2023-04-16 20:53:46,583 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6e3ca267fe05482c84c67369e4d738a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681678424334 2023-04-16 20:53:46,584 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6f4fef4a71814b4ba08cf9c9ee16c620, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681678425513 2023-04-16 20:53:46,585 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e497b7d46e2a4e61bc24c1ee5e33206b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681678425853 2023-04-16 20:53:46,590 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:46,592 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:46,600 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,601 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,601 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:46,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:46,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12461884[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:46,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:46,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678486584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,610 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678486585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:46,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:46,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:46,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678486603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,611 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,611 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:46,612 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678486607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,615 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#28 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:46,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678486609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:46,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:46,627 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:46,627 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:46,627 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:46,627 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:46,628 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:46,630 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,630 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,630 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,632 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,639 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,639 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,641 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:46,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:46,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:46,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,641 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:46,641 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:46,666 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/825fbe89ca634aedae01c63c9e456d36 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/825fbe89ca634aedae01c63c9e456d36 2023-04-16 20:53:46,677 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into 825fbe89ca634aedae01c63c9e456d36(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:46,677 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:46,677 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=11, startTime=1681678426464; duration=0sec 2023-04-16 20:53:46,677 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:46,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678486711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678486712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678486712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678486724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678486724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,741 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:46,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:46,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:46,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:46,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:46,750 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:46,750 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,751 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:46,751 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:46,751 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,751 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,753 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:46,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:46,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:46,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:46,756 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,756 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:46,756 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:46,756 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:46,765 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:46,768 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:46,768 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b48b4f4[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:46,774 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:46,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:46,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:46,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:46,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:46,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:46,777 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:46,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:46,782 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:46,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:46,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:46,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:46,782 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:46,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,785 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,790 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,790 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,790 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:46,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:46,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,791 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:46,791 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:46,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:46,891 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:46,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:46,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:46,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:46,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:46,904 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:46,904 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:46,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:46,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,909 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:46,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:46,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:46,921 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,921 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:46,922 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:46,922 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:46,924 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678486922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:46,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678486923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678486923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:46,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725d0fd5[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:46,928 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:46,928 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:46,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:46,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678486929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:46,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678486930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:46,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:46,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:46,938 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:46,938 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:46,938 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:46,938 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:46,938 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:46,944 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,944 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,944 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:46,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:46,945 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,949 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,954 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/2712c722476e4050ae6703f1c5a1dbf0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/2712c722476e4050ae6703f1c5a1dbf0 2023-04-16 20:53:46,958 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:46,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:46,958 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:46,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:46,959 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,959 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,960 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:46,961 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:46,961 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,961 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:46,961 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:46,961 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:46,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:46,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:46,966 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:46,966 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:46,975 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into 2712c722476e4050ae6703f1c5a1dbf0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:46,975 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:46,975 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=11, startTime=1681678426463; duration=0sec 2023-04-16 20:53:46,975 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:47,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/1962257b0d3a48e4b6990f9a06a54a6a 2023-04-16 20:53:47,066 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:47,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:47,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:47,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:47,078 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:47,078 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:47,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:47,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,084 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:47,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:47,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:47,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:47,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:47,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:47,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:47,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:47,089 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,089 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:47,089 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:47,089 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:47,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:47,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:47,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:47,091 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:47,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:47,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b987105[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:47,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:47,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,103 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:47,104 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,104 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:47,107 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:47,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:47,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:47,118 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:47,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:47,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:47,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:47,118 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:47,120 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,120 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,120 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,121 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,126 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,126 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,127 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,127 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:47,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:47,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:47,129 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:47,127 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,129 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,129 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,129 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,129 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:47,229 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:47,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:47,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678487229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678487229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678487230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:47,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678487234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:47,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:47,234 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:47,235 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:47,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:47,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678487234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,239 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,239 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,240 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:47,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:47,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:47,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:47,241 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:47,242 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,242 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:47,242 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:47,243 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:47,244 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,245 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:47,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b8dcca[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:47,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:47,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,264 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,264 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:47,264 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:47,264 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:47,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:47,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:47,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:47,271 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:47,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:47,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:47,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:47,272 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:47,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,278 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,281 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,287 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,287 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,289 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,289 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,289 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,289 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:47,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,289 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,291 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:47,291 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:47,391 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:47,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:47,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:47,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:47,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:47,397 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:47,397 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:47,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:47,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,400 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,400 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,401 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,401 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:47,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:47,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:47,403 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:47,404 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:47,404 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,404 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:47,404 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:47,405 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,405 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:47,405 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:47,405 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,405 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:47,405 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:47,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,410 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,411 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,411 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:47,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7739241d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:47,416 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:47,416 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,418 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,418 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:47,418 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:47,418 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:47,418 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,418 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:47,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:47,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,419 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:47,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:47,430 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:47,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:47,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:47,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:47,430 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:47,432 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,432 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,432 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,435 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,446 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:47,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:47,448 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:47,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/fccc3c12122c47aba70abf3716e7954f 2023-04-16 20:53:47,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/be19d543fc5f4077b89c906df5b71870 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/be19d543fc5f4077b89c906df5b71870 2023-04-16 20:53:47,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/be19d543fc5f4077b89c906df5b71870, entries=200, sequenceid=149, filesize=14.1 K 2023-04-16 20:53:47,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/1962257b0d3a48e4b6990f9a06a54a6a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1962257b0d3a48e4b6990f9a06a54a6a 2023-04-16 20:53:47,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1962257b0d3a48e4b6990f9a06a54a6a, entries=150, sequenceid=149, filesize=11.8 K 2023-04-16 20:53:47,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/fccc3c12122c47aba70abf3716e7954f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/fccc3c12122c47aba70abf3716e7954f 2023-04-16 20:53:47,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/fccc3c12122c47aba70abf3716e7954f, entries=150, sequenceid=149, filesize=11.8 K 2023-04-16 20:53:47,514 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for bfe0529baac322a72dbf6ef88dd5ff1c in 968ms, sequenceid=149, compaction requested=false 2023-04-16 20:53:47,515 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:47,548 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:47,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:47,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:47,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:47,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:47,556 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:47,556 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:47,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:47,558 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,558 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,559 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,559 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,560 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,560 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:47,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:47,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:47,565 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,566 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:47,566 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-16 20:53:47,566 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:47,566 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:47,566 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:47,566 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:47,567 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:47,567 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:47,582 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/ecf049a0540645faa43f4614bb24b802 2023-04-16 20:53:47,612 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/d43d9c5d66634f20b4c5918f47d8c84c 2023-04-16 20:53:47,632 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/8b64791bfecd4e8fb74e02af65022556 2023-04-16 20:53:47,641 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/ecf049a0540645faa43f4614bb24b802 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ecf049a0540645faa43f4614bb24b802 2023-04-16 20:53:47,651 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ecf049a0540645faa43f4614bb24b802, entries=150, sequenceid=168, filesize=11.8 K 2023-04-16 20:53:47,656 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/d43d9c5d66634f20b4c5918f47d8c84c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d43d9c5d66634f20b4c5918f47d8c84c 2023-04-16 20:53:47,672 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d43d9c5d66634f20b4c5918f47d8c84c, entries=150, sequenceid=168, filesize=11.8 K 2023-04-16 20:53:47,675 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/8b64791bfecd4e8fb74e02af65022556 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/8b64791bfecd4e8fb74e02af65022556 2023-04-16 20:53:47,686 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/8b64791bfecd4e8fb74e02af65022556, entries=150, sequenceid=168, filesize=11.8 K 2023-04-16 20:53:47,689 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for bfe0529baac322a72dbf6ef88dd5ff1c in 122ms, sequenceid=168, compaction requested=true 2023-04-16 20:53:47,689 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:47,689 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,689 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:47,705 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,707 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:47,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d00ae88[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:47,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,710 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,710 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:47,710 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:47,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:47,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:47,730 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:47,731 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,731 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:47,731 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:47,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:47,736 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:47,736 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:47,736 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:47,736 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:47,736 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:47,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,742 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,745 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,748 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,750 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:47,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:47,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,753 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,753 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,753 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,753 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,753 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,753 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:53:47,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:47,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,756 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:47,756 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:47,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:47,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:47,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:47,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:47,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:47,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:47,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/57a54635217d41a4962b07af89c3acf6 2023-04-16 20:53:47,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678427772, max timestamp=1681678427797Num uniques 859; 2023-04-16 20:53:47,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678427772, max timestamp=1681678427797Num uniques 859; 2023-04-16 20:53:47,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:53:47,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:47,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678427772, max timestamp=1681678427797Num uniques 859; 2023-04-16 20:53:47,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:47,856 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:47,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:47,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:47,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678487856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:47,862 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:47,862 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,863 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:47,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678487856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,863 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:47,864 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,864 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678487857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678487857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678487858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,866 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,866 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,866 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:47,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:47,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:47,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:47,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:47,885 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:47,885 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:47,885 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:47,885 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,885 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:47,885 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:47,885 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:47,888 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,888 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,889 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,889 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:47,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1568ea74[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:47,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:47,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:47,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:47,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:47,898 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,898 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:47,899 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:47,899 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,899 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:47,901 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:47,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:47,905 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:47,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:47,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:47,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:47,905 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:47,910 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,910 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,911 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:47,912 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,914 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:47,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:47,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:47,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:47,918 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,918 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:47,918 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:47,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:47,918 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:47,921 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:47,921 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:47,921 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:47,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:47,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678487964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678487965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678487966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678487967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:47,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:47,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678487967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,021 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:48,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:48,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:48,034 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:48,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:48,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:48,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,039 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:48,039 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,040 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,040 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:48,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:48,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:48,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:48,051 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,051 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:48,051 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:48,051 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:48,053 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,053 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,054 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:48,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae93a9f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:48,055 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:48,055 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:48,058 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,058 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:48,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:48,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:48,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:48,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:48,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:48,065 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:48,065 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:48,065 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:48,065 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:48,065 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:48,068 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,068 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,068 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,069 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,071 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,074 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:48,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:48,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:48,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:48,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:48,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:48,075 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:48,076 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:48,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:48,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:48,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678488169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678488169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,176 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:48,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678488174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:48,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678488174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678488176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:48,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:48,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:48,185 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:48,185 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:48,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:48,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,187 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,189 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:48,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:48,190 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,190 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:48,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:48,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:48,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:48,191 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:48,192 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,194 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:48,194 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:48,194 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:48,195 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,195 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:48,195 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:48,195 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:48,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:48,199 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,200 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744ef613[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:48,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:48,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,203 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:48,204 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,204 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:48,205 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:48,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:48,216 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:48,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:48,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:48,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:48,216 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:48,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:48,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,222 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/af0a4d273ac9452db751ff6387132e05 2023-04-16 20:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,223 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,228 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:48,228 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:48,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:48,228 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:48,228 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:48,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:48,228 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,231 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,230 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:48,231 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:48,231 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,231 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:48,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:48,244 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/24a6f3cd439e40f786ee255e5d02c0e5 2023-04-16 20:53:48,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/57a54635217d41a4962b07af89c3acf6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/57a54635217d41a4962b07af89c3acf6 2023-04-16 20:53:48,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/57a54635217d41a4962b07af89c3acf6, entries=200, sequenceid=179, filesize=14.1 K 2023-04-16 20:53:48,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/af0a4d273ac9452db751ff6387132e05 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/af0a4d273ac9452db751ff6387132e05 2023-04-16 20:53:48,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/af0a4d273ac9452db751ff6387132e05, entries=150, sequenceid=179, filesize=11.8 K 2023-04-16 20:53:48,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/24a6f3cd439e40f786ee255e5d02c0e5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/24a6f3cd439e40f786ee255e5d02c0e5 2023-04-16 20:53:48,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/24a6f3cd439e40f786ee255e5d02c0e5, entries=150, sequenceid=179, filesize=11.8 K 2023-04-16 20:53:48,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for bfe0529baac322a72dbf6ef88dd5ff1c in 531ms, sequenceid=179, compaction requested=true 2023-04-16 20:53:48,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:48,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:48,284 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:48,285 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:48,284 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:48,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:48,288 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53148 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:48,288 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:48,288 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,289 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/137d9bf6306d4179a7991bb6ce3acbfb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/be19d543fc5f4077b89c906df5b71870, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ecf049a0540645faa43f4614bb24b802, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/57a54635217d41a4962b07af89c3acf6] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=51.9 K 2023-04-16 20:53:48,290 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 137d9bf6306d4179a7991bb6ce3acbfb, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681678425853 2023-04-16 20:53:48,290 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting be19d543fc5f4077b89c906df5b71870, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681678426210 2023-04-16 20:53:48,290 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48368 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:48,292 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:48,292 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,292 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/2712c722476e4050ae6703f1c5a1dbf0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1962257b0d3a48e4b6990f9a06a54a6a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d43d9c5d66634f20b4c5918f47d8c84c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/af0a4d273ac9452db751ff6387132e05] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=47.2 K 2023-04-16 20:53:48,293 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ecf049a0540645faa43f4614bb24b802, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681678426600 2023-04-16 20:53:48,294 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 57a54635217d41a4962b07af89c3acf6, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681678427744 2023-04-16 20:53:48,294 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2712c722476e4050ae6703f1c5a1dbf0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681678425853 2023-04-16 20:53:48,300 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1962257b0d3a48e4b6990f9a06a54a6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681678426215 2023-04-16 20:53:48,301 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d43d9c5d66634f20b4c5918f47d8c84c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681678426600 2023-04-16 20:53:48,301 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting af0a4d273ac9452db751ff6387132e05, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681678427744 2023-04-16 20:53:48,317 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#36 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:48,323 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#37 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:48,331 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:48,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:48,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:48,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:48,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:48,338 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:48,338 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,338 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:48,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:48,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,341 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:48,341 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,341 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,342 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:48,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:48,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:48,349 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,349 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:48,350 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-04-16 20:53:48,350 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:48,350 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:48,350 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:48,350 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:48,350 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:48,350 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:48,359 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/eb866f34ae2848ba9d89b653b4906dff as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/eb866f34ae2848ba9d89b653b4906dff 2023-04-16 20:53:48,366 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into eb866f34ae2848ba9d89b653b4906dff(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:48,366 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:48,366 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=12, startTime=1681678428284; duration=0sec 2023-04-16 20:53:48,367 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:48,367 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:48,369 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48368 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:48,369 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:48,369 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,369 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/825fbe89ca634aedae01c63c9e456d36, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/fccc3c12122c47aba70abf3716e7954f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/8b64791bfecd4e8fb74e02af65022556, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/24a6f3cd439e40f786ee255e5d02c0e5] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=47.2 K 2023-04-16 20:53:48,371 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/6660950e49394dac87fc6769f3eafbf0 2023-04-16 20:53:48,376 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 825fbe89ca634aedae01c63c9e456d36, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681678425853 2023-04-16 20:53:48,377 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting fccc3c12122c47aba70abf3716e7954f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681678426215 2023-04-16 20:53:48,377 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8b64791bfecd4e8fb74e02af65022556, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681678426600 2023-04-16 20:53:48,378 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 24a6f3cd439e40f786ee255e5d02c0e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681678427744 2023-04-16 20:53:48,390 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#39 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:48,409 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/810fa7544b724eac8d90abd6ed3ceb2a 2023-04-16 20:53:48,419 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/2236a062809041f0929630733cc051fd as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2236a062809041f0929630733cc051fd 2023-04-16 20:53:48,429 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into 2236a062809041f0929630733cc051fd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:48,429 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:48,429 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=12, startTime=1681678428285; duration=0sec 2023-04-16 20:53:48,429 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:48,447 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/dc6c95c8dabc4918b6f15d201bf5d9fc 2023-04-16 20:53:48,461 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/6660950e49394dac87fc6769f3eafbf0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6660950e49394dac87fc6769f3eafbf0 2023-04-16 20:53:48,472 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6660950e49394dac87fc6769f3eafbf0, entries=150, sequenceid=215, filesize=11.8 K 2023-04-16 20:53:48,476 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/810fa7544b724eac8d90abd6ed3ceb2a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/810fa7544b724eac8d90abd6ed3ceb2a 2023-04-16 20:53:48,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:48,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:48,508 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/810fa7544b724eac8d90abd6ed3ceb2a, entries=150, sequenceid=215, filesize=11.8 K 2023-04-16 20:53:48,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/dc6c95c8dabc4918b6f15d201bf5d9fc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dc6c95c8dabc4918b6f15d201bf5d9fc 2023-04-16 20:53:48,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678488506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678488509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678488517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678488519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,526 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dc6c95c8dabc4918b6f15d201bf5d9fc, entries=150, sequenceid=215, filesize=11.8 K 2023-04-16 20:53:48,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678488511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,527 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for bfe0529baac322a72dbf6ef88dd5ff1c in 177ms, sequenceid=215, compaction requested=false 2023-04-16 20:53:48,527 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:48,527 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,527 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:48,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,530 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,531 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,531 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:48,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c5262fc[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:48,535 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:48,535 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,543 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:48,544 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,544 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:48,546 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:48,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:48,551 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:48,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:48,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:48,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:48,551 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:48,553 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,553 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,554 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:48,554 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,556 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,570 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:48,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:48,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:48,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:48,572 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:48,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:48,572 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:48,572 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:48,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:48,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:48,625 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:53:48,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:48,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:48,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:48,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:48,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:48,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:48,672 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:48,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:48,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:48,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678488663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678488665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678488667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678488681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678488689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:48,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:48,696 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:48,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:48,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:48,697 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,697 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:48,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:48,705 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,705 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:48,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:48,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:48,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:48,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:48,721 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:48,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,721 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:48,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:48,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:48,721 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/f376b60ebedf491e97c886ba8088e158 2023-04-16 20:53:48,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,721 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:48,724 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e3b1048[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:48,731 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:48,731 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,739 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,739 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,739 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:48,744 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,744 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:48,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:48,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:48,757 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:48,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:48,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:48,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:48,757 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:48,763 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,763 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:48,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,764 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/f4ea32898f504c24b4d1896fc4e97104 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f4ea32898f504c24b4d1896fc4e97104 2023-04-16 20:53:48,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:48,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:48,770 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:48,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:48,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:48,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:48,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:48,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:48,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:48,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,773 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:48,773 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:48,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:48,776 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into f4ea32898f504c24b4d1896fc4e97104(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:48,776 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:48,776 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=12, startTime=1681678428284; duration=0sec 2023-04-16 20:53:48,776 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:48,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:48,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/468b07659f5143f292d313b2aea6c688 2023-04-16 20:53:48,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678488783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678488783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678488783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678488791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:48,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678488796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/7cebae6b18dd44bc91f5c93959b4ba78 2023-04-16 20:53:48,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/f376b60ebedf491e97c886ba8088e158 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f376b60ebedf491e97c886ba8088e158 2023-04-16 20:53:48,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f376b60ebedf491e97c886ba8088e158, entries=150, sequenceid=229, filesize=11.8 K 2023-04-16 20:53:48,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/468b07659f5143f292d313b2aea6c688 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/468b07659f5143f292d313b2aea6c688 2023-04-16 20:53:48,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/468b07659f5143f292d313b2aea6c688, entries=150, sequenceid=229, filesize=11.8 K 2023-04-16 20:53:48,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/7cebae6b18dd44bc91f5c93959b4ba78 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7cebae6b18dd44bc91f5c93959b4ba78 2023-04-16 20:53:48,874 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:48,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:48,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7cebae6b18dd44bc91f5c93959b4ba78, entries=150, sequenceid=229, filesize=11.8 K 2023-04-16 20:53:48,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:48,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bfe0529baac322a72dbf6ef88dd5ff1c in 255ms, sequenceid=229, compaction requested=true 2023-04-16 20:53:48,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:48,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:48,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:53:48,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-16 20:53:48,881 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:48,882 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:48,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:48,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:48,882 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:48,882 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,882 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:48,882 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:48,883 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:48,883 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:48,883 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:48,883 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,884 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:48,884 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,884 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,883 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,884 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f4ea32898f504c24b4d1896fc4e97104, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6660950e49394dac87fc6769f3eafbf0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f376b60ebedf491e97c886ba8088e158] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=35.7 K 2023-04-16 20:53:48,884 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2236a062809041f0929630733cc051fd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dc6c95c8dabc4918b6f15d201bf5d9fc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7cebae6b18dd44bc91f5c93959b4ba78] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=35.7 K 2023-04-16 20:53:48,884 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting f4ea32898f504c24b4d1896fc4e97104, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681678427744 2023-04-16 20:53:48,897 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 2236a062809041f0929630733cc051fd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681678427744 2023-04-16 20:53:48,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,901 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:48,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:48,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:48,901 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6660950e49394dac87fc6769f3eafbf0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681678427844 2023-04-16 20:53:48,901 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting dc6c95c8dabc4918b6f15d201bf5d9fc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681678427844 2023-04-16 20:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:48,902 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting f376b60ebedf491e97c886ba8088e158, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681678428507 2023-04-16 20:53:48,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:48,903 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7cebae6b18dd44bc91f5c93959b4ba78, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681678428507 2023-04-16 20:53:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:48,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:48,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:48,907 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,907 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:48,908 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 20:53:48,915 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:48,916 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:48,916 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:48,916 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:48,916 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:48,916 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:48,928 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#45 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:48,933 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:48,968 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/211ef114d2b54bcfa977adfff9074761 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/211ef114d2b54bcfa977adfff9074761 2023-04-16 20:53:48,971 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/65f6889460a640f284149f8d45991b12 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/65f6889460a640f284149f8d45991b12 2023-04-16 20:53:48,978 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into 211ef114d2b54bcfa977adfff9074761(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:48,978 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:48,978 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=13, startTime=1681678428881; duration=0sec 2023-04-16 20:53:48,978 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:48,978 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:48,982 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into 65f6889460a640f284149f8d45991b12(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:48,983 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:48,983 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=13, startTime=1681678428881; duration=0sec 2023-04-16 20:53:48,983 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:48,983 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:48,983 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:48,983 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:48,984 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/eb866f34ae2848ba9d89b653b4906dff, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/810fa7544b724eac8d90abd6ed3ceb2a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/468b07659f5143f292d313b2aea6c688] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=35.7 K 2023-04-16 20:53:48,987 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting eb866f34ae2848ba9d89b653b4906dff, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681678427744 2023-04-16 20:53:48,988 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 810fa7544b724eac8d90abd6ed3ceb2a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681678427844 2023-04-16 20:53:48,988 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 468b07659f5143f292d313b2aea6c688, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681678428507 2023-04-16 20:53:48,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:48,998 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:49,004 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#48 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:49,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678489003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678489004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678489005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678489007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678489006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,018 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/f6dfd54efd3d49dbb5ece086d8cf6d2d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6dfd54efd3d49dbb5ece086d8cf6d2d 2023-04-16 20:53:49,025 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into f6dfd54efd3d49dbb5ece086d8cf6d2d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:49,026 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:49,026 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=13, startTime=1681678428881; duration=0sec 2023-04-16 20:53:49,026 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:49,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678489112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678489112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678489112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678489112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678489115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678489315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678489316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678489316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678489317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678489319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,357 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/e5b4e9cbb58040e8a7274fb7cfcf59c3 2023-04-16 20:53:49,381 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/aa5621fac86644eca776883904001a95 2023-04-16 20:53:49,420 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/40e31a6608e3474daeeb3ca9ddb12e2c 2023-04-16 20:53:49,430 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/e5b4e9cbb58040e8a7274fb7cfcf59c3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/e5b4e9cbb58040e8a7274fb7cfcf59c3 2023-04-16 20:53:49,444 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/e5b4e9cbb58040e8a7274fb7cfcf59c3, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 20:53:49,445 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/aa5621fac86644eca776883904001a95 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aa5621fac86644eca776883904001a95 2023-04-16 20:53:49,451 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aa5621fac86644eca776883904001a95, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 20:53:49,453 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/40e31a6608e3474daeeb3ca9ddb12e2c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/40e31a6608e3474daeeb3ca9ddb12e2c 2023-04-16 20:53:49,458 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/40e31a6608e3474daeeb3ca9ddb12e2c, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 20:53:49,459 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bfe0529baac322a72dbf6ef88dd5ff1c in 551ms, sequenceid=250, compaction requested=false 2023-04-16 20:53:49,459 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:49,459 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:49,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:49,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:49,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:49,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:49,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,461 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:49,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:49,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:49,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d85e6d2[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:49,467 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:49,467 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,468 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,468 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:49,468 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:49,468 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:49,468 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,468 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:49,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:49,470 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:49,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:49,474 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:49,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:49,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:49,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:49,474 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:49,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:49,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,480 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:49,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:49,493 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:49,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:49,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:49,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:49,493 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:49,493 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:49,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:49,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:49,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:49,594 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:49,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:49,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:49,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:49,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:49,600 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:49,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:49,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:49,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:49,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,602 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:49,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:49,602 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,602 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:49,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:49,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:49,603 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:49,604 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:49,604 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:49,604 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:53:49,605 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:49,605 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:49,605 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:49,605 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:49,605 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:49,605 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:49,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:49,621 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:49,625 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/3a8c50bd92984fa3b57c4a4625890690 2023-04-16 20:53:49,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43166 deadline: 1681678489650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43226 deadline: 1681678489652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678489652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43196 deadline: 1681678489659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43186 deadline: 1681678489660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,677 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/b70d4593eceb474c949169ff207f4c57 2023-04-16 20:53:49,713 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/a1ae1df1c81a4bd79f461a392f70e211 2023-04-16 20:53:49,725 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/3a8c50bd92984fa3b57c4a4625890690 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/3a8c50bd92984fa3b57c4a4625890690 2023-04-16 20:53:49,733 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/3a8c50bd92984fa3b57c4a4625890690, entries=150, sequenceid=265, filesize=11.9 K 2023-04-16 20:53:49,734 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/b70d4593eceb474c949169ff207f4c57 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/b70d4593eceb474c949169ff207f4c57 2023-04-16 20:53:49,741 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/b70d4593eceb474c949169ff207f4c57, entries=150, sequenceid=265, filesize=11.9 K 2023-04-16 20:53:49,742 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/a1ae1df1c81a4bd79f461a392f70e211 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a1ae1df1c81a4bd79f461a392f70e211 2023-04-16 20:53:49,748 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a1ae1df1c81a4bd79f461a392f70e211, entries=150, sequenceid=265, filesize=11.9 K 2023-04-16 20:53:49,750 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bfe0529baac322a72dbf6ef88dd5ff1c in 146ms, sequenceid=265, compaction requested=true 2023-04-16 20:53:49,750 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:49,750 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:49,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:49,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:49,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:49,758 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,759 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:49,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e786f1f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:49,761 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:49,761 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,763 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:49,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:49,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:49,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:49,766 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:49,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:49,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:49,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:49,766 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:49,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:49,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:49,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:49,769 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,771 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:49,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-04-16 20:53:49,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678429650, max timestamp=1681678429771Num uniques 859; 2023-04-16 20:53:49,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:49,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678429650, max timestamp=1681678429769Num uniques 859; 2023-04-16 20:53:49,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:49,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678429650, max timestamp=1681678429770Num uniques 859; 2023-04-16 20:53:49,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:49,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:49,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:49,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:49,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:49,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:49,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:49,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:49,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:49,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:49,779 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:49,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:49,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:49,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,779 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:49,779 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:49,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:49,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:49,820 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/22bec2b28f80444e841dd6db7ee51ea0 2023-04-16 20:53:49,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678429809, max timestamp=1681678429813Num uniques 859; 2023-04-16 20:53:49,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:49,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678429809, max timestamp=1681678429813Num uniques 859; 2023-04-16 20:53:49,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:49,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678429809, max timestamp=1681678429813Num uniques 859; 2023-04-16 20:53:49,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 20:53:49,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681678429813, max timestamp=1681678429864Num uniques 1009; 2023-04-16 20:53:49,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-16 20:53:49,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681678429813, max timestamp=1681678429864Num uniques 1018; 2023-04-16 20:53:49,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-16 20:53:49,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681678429813, max timestamp=1681678429864Num uniques 1018; 2023-04-16 20:53:49,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-16 20:53:49,879 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:49,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:49,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:49,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:49,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:49,900 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:49,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:49,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:49,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:49,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,902 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:49,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:49,902 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,903 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,903 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:49,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:49,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:49,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:49,931 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:49,931 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:49,931 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:49,931 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:49,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43174 deadline: 1681678489912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681678429903, max timestamp=1681678429915Num uniques 1077; 2023-04-16 20:53:49,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-16 20:53:49,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681678429903, max timestamp=1681678429917Num uniques 1077; 2023-04-16 20:53:49,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-16 20:53:49,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681678429903, max timestamp=1681678429917Num uniques 1077; 2023-04-16 20:53:49,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-16 20:53:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,941 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:49,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:49,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41b2b1ae[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:49,944 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:49,944 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,950 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,950 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:49,950 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:49,950 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:49,950 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,950 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:49,952 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:49,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:49,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:49,968 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:49,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:49,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:49,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:49,968 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:49,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:49,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:49,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,981 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,984 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:49,984 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:49,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:49,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:49,984 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,984 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,984 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:49,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:49,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:49,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:49,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:49,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:49,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:49,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:49,985 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:49,985 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:49,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678429970, max timestamp=1681678429986Num uniques 1036; 2023-04-16 20:53:49,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-16 20:53:49,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678429970, max timestamp=1681678429986Num uniques 1036; 2023-04-16 20:53:49,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-16 20:53:49,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681678429951, max timestamp=1681678429986Num uniques 1036; 2023-04-16 20:53:49,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-16 20:53:50,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681678430006, max timestamp=1681678430029Num uniques 1195; 2023-04-16 20:53:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=13510, suffix heap size=152248,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=295 2023-04-16 20:53:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681678430006, max timestamp=1681678430029Num uniques 1045; 2023-04-16 20:53:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-16 20:53:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13500, totalHeapSize=30920, min timestamp=1681678430006, max timestamp=1681678430029Num uniques 1045; 2023-04-16 20:53:50,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=13500, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-16 20:53:50,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678490076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678490076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678490077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678490078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678490079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,085 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:50,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:50,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:50,092 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:50,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:50,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:50,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,096 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,096 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:50,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:50,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:50,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:50,100 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,100 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:50,100 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:50,100 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:50,102 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:50,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@579d5080[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:50,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:50,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,106 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,106 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:50,106 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:50,106 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:50,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,107 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:50,108 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,108 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,108 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,109 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:50,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:50,113 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:50,113 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:50,113 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:50,113 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:50,113 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:50,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,117 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,121 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,121 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,121 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,121 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,121 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,121 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:50,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:50,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:50,121 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,122 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,122 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,122 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,122 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:50,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,122 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:50,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678490184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678490185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678490186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678490186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678490188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,222 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:50,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:50,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:50,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:50,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:50,244 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:50,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:50,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:50,245 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,245 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,247 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,247 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,247 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:50,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:50,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:50,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/d2f3355d5e754b9a97d18cec037e0091 2023-04-16 20:53:50,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:50,272 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,272 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:50,272 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:50,272 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:50,274 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:50,274 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:50,274 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,274 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:50,274 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:50,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35cc8fd7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:50,279 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:50,279 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,280 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,280 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,280 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:50,283 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:50,287 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:50,287 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:50,287 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:50,287 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:50,287 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:50,289 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,289 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,289 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/93ca639ddd204b098d0ebf2f94f88158 2023-04-16 20:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,291 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,293 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,296 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:50,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:50,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,296 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:50,297 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:50,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:50,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/22bec2b28f80444e841dd6db7ee51ea0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/22bec2b28f80444e841dd6db7ee51ea0 2023-04-16 20:53:50,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/22bec2b28f80444e841dd6db7ee51ea0, entries=150, sequenceid=288, filesize=11.9 K 2023-04-16 20:53:50,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/d2f3355d5e754b9a97d18cec037e0091 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d2f3355d5e754b9a97d18cec037e0091 2023-04-16 20:53:50,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d2f3355d5e754b9a97d18cec037e0091, entries=150, sequenceid=288, filesize=11.9 K 2023-04-16 20:53:50,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/93ca639ddd204b098d0ebf2f94f88158 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/93ca639ddd204b098d0ebf2f94f88158 2023-04-16 20:53:50,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/93ca639ddd204b098d0ebf2f94f88158, entries=150, sequenceid=288, filesize=11.9 K 2023-04-16 20:53:50,332 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=147.60 KB/151140 for bfe0529baac322a72dbf6ef88dd5ff1c in 558ms, sequenceid=288, compaction requested=true 2023-04-16 20:53:50,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:50,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:50,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:50,333 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:50,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:50,333 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:50,334 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:50,335 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:50,335 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:50,335 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,335 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/211ef114d2b54bcfa977adfff9074761, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/e5b4e9cbb58040e8a7274fb7cfcf59c3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/3a8c50bd92984fa3b57c4a4625890690, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/22bec2b28f80444e841dd6db7ee51ea0] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=47.8 K 2023-04-16 20:53:50,335 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:50,335 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,335 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6dfd54efd3d49dbb5ece086d8cf6d2d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aa5621fac86644eca776883904001a95, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/b70d4593eceb474c949169ff207f4c57, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d2f3355d5e754b9a97d18cec037e0091] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=47.8 K 2023-04-16 20:53:50,336 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 211ef114d2b54bcfa977adfff9074761, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681678428507 2023-04-16 20:53:50,336 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting f6dfd54efd3d49dbb5ece086d8cf6d2d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681678428507 2023-04-16 20:53:50,336 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e5b4e9cbb58040e8a7274fb7cfcf59c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681678428663 2023-04-16 20:53:50,336 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting aa5621fac86644eca776883904001a95, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681678428663 2023-04-16 20:53:50,337 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3a8c50bd92984fa3b57c4a4625890690, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678429001 2023-04-16 20:53:50,337 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 22bec2b28f80444e841dd6db7ee51ea0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681678429657 2023-04-16 20:53:50,337 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b70d4593eceb474c949169ff207f4c57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678429001 2023-04-16 20:53:50,344 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d2f3355d5e754b9a97d18cec037e0091, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681678429657 2023-04-16 20:53:50,359 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:50,361 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#58 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:50,386 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/fb39135d43c74befb41bf8ddf67ab2d6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fb39135d43c74befb41bf8ddf67ab2d6 2023-04-16 20:53:50,397 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:50,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:50,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=154.31 KB heapSize=432 KB 2023-04-16 20:53:50,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:50,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681678430074, max timestamp=1681678430397Num uniques 1154; 2023-04-16 20:53:50,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:50,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-16 20:53:50,401 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into fb39135d43c74befb41bf8ddf67ab2d6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:50,401 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:50,401 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=12, startTime=1681678430333; duration=0sec 2023-04-16 20:53:50,401 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:50,401 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:50,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681678430074, max timestamp=1681678430397Num uniques 1154; 2023-04-16 20:53:50,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-16 20:53:50,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681678430074, max timestamp=1681678430397Num uniques 1154; 2023-04-16 20:53:50,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-16 20:53:50,403 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:50,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:50,403 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:50,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:50,403 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,404 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:50,404 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:50,404 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:50,404 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:50,404 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/65f6889460a640f284149f8d45991b12, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/40e31a6608e3474daeeb3ca9ddb12e2c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a1ae1df1c81a4bd79f461a392f70e211, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/93ca639ddd204b098d0ebf2f94f88158] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=47.8 K 2023-04-16 20:53:50,406 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 65f6889460a640f284149f8d45991b12, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681678428507 2023-04-16 20:53:50,407 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 40e31a6608e3474daeeb3ca9ddb12e2c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681678428663 2023-04-16 20:53:50,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:50,408 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:50,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:50,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:50,408 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting a1ae1df1c81a4bd79f461a392f70e211, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678429001 2023-04-16 20:53:50,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,409 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,409 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 93ca639ddd204b098d0ebf2f94f88158, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681678429657 2023-04-16 20:53:50,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,411 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,411 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:50,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:50,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:50,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:50,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:50,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:50,420 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,420 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:50,420 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:50,420 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,421 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,428 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/a8d8f6fca5e6471ebc5e5083b75750f9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a8d8f6fca5e6471ebc5e5083b75750f9 2023-04-16 20:53:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:50,438 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into a8d8f6fca5e6471ebc5e5083b75750f9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:50,438 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:50,438 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=12, startTime=1681678430333; duration=0sec 2023-04-16 20:53:50,438 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:50,442 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#60 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:50,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,445 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c11ee6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:50,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:50,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,449 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,449 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:50,449 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:50,449 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:50,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:50,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:50,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:50,454 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:50,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:50,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:50,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:50,455 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:50,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,459 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/db57cd3498b3406dbed04fc2a466fd1e 2023-04-16 20:53:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678430426, max timestamp=1681678430458Num uniques 859; 2023-04-16 20:53:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:53:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678430432, max timestamp=1681678430460Num uniques 859; 2023-04-16 20:53:50,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:50,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678430432, max timestamp=1681678430460Num uniques 859; 2023-04-16 20:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:50,463 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,468 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,468 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,468 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,468 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,468 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,468 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:50,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:50,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,469 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:50,470 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:50,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681678430482, max timestamp=1681678430494Num uniques 1018; 2023-04-16 20:53:50,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681678430482, max timestamp=1681678430494Num uniques 1018; 2023-04-16 20:53:50,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-16 20:53:50,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-16 20:53:50,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681678430482, max timestamp=1681678430493Num uniques 1068; 2023-04-16 20:53:50,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-16 20:53:50,506 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/1afc3e0b355d4cbfae26175cfadf2ee2 2023-04-16 20:53:50,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/3033d918c6d84c3ab7e7ab635206f628 2023-04-16 20:53:50,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681678430508, max timestamp=1681678430529Num uniques 1077; 2023-04-16 20:53:50,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=17262, suffix heap size=139976,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=377 2023-04-16 20:53:50,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681678430508, max timestamp=1681678430529Num uniques 1077; 2023-04-16 20:53:50,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=17262, suffix heap size=139976,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=377 2023-04-16 20:53:50,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681678430508, max timestamp=1681678430529Num uniques 1077; 2023-04-16 20:53:50,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=17262, suffix heap size=139976,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=377 2023-04-16 20:53:50,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/db57cd3498b3406dbed04fc2a466fd1e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/db57cd3498b3406dbed04fc2a466fd1e 2023-04-16 20:53:50,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/db57cd3498b3406dbed04fc2a466fd1e, entries=150, sequenceid=396, filesize=11.9 K 2023-04-16 20:53:50,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/1afc3e0b355d4cbfae26175cfadf2ee2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1afc3e0b355d4cbfae26175cfadf2ee2 2023-04-16 20:53:50,570 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:50,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:50,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1afc3e0b355d4cbfae26175cfadf2ee2, entries=150, sequenceid=396, filesize=11.9 K 2023-04-16 20:53:50,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:50,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/3033d918c6d84c3ab7e7ab635206f628 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3033d918c6d84c3ab7e7ab635206f628 2023-04-16 20:53:50,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678430572, max timestamp=1681678430575Num uniques 1236; 2023-04-16 20:53:50,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=8516, suffix heap size=156512,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=186 2023-04-16 20:53:50,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678430572, max timestamp=1681678430575Num uniques 1236; 2023-04-16 20:53:50,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678430572, max timestamp=1681678430575Num uniques 1236; 2023-04-16 20:53:50,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=8516, suffix heap size=156512,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=186 2023-04-16 20:53:50,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=8516, suffix heap size=156512,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=186 2023-04-16 20:53:50,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:50,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:50,580 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:50,580 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,580 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:50,580 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:50,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,582 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,582 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3033d918c6d84c3ab7e7ab635206f628, entries=150, sequenceid=396, filesize=11.9 K 2023-04-16 20:53:50,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:50,585 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~81.89 KB/83856, currentSize=33.54 KB/34350 for bfe0529baac322a72dbf6ef88dd5ff1c in 187ms, sequenceid=396, compaction requested=false 2023-04-16 20:53:50,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:50,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:50,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=53.67 KB heapSize=143.39 KB 2023-04-16 20:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:50,594 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,594 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:50,594 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:50,594 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:50,596 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:50,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c2ae791[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:50,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:50,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,601 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,602 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:50,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:50,605 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:50,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:50,610 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:50,610 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:50,610 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:50,610 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:50,610 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:50,611 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,611 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,611 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,612 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,614 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/8c3854c29d5c40b7a68ea6daa4f562de 2023-04-16 20:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:50,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:50,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:50,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,624 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:50,624 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,625 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:50,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681678430616, max timestamp=1681678430640Num uniques 859; 2023-04-16 20:53:50,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-16 20:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681678430616, max timestamp=1681678430640Num uniques 859; 2023-04-16 20:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-16 20:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681678430616, max timestamp=1681678430640Num uniques 859; 2023-04-16 20:53:50,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-16 20:53:50,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/fc10ea6f413946e4bf15533fffde5e0d 2023-04-16 20:53:50,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678490652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678490653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678490658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678490659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678490661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,725 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:50,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:50,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:50,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:50,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:50,732 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:50,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:50,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:50,733 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,733 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,734 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:50,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:50,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:50,741 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,741 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:50,741 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:50,741 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:50,744 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,744 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,745 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,745 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:50,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@510eb3c4[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:50,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:50,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:50,748 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,748 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:50,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:50,753 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:50,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:50,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:50,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:50,753 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:50,756 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,756 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,756 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,762 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,762 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,762 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,762 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,763 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,763 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,763 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:50,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:50,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:50,763 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,763 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:50,763 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:50,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678490762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678490762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678490763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678490765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678490765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,863 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:50,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:50,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:50,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:50,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:50,869 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:50,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:50,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:50,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,871 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,871 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,871 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:50,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:50,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:50,874 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,874 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:50,874 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:50,874 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:50,876 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:50,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed2c05[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:50,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:50,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,883 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:50,883 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:50,883 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:50,883 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,883 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:50,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:50,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:50,888 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:50,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:50,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:50,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:50,888 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:50,890 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,890 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,890 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,891 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,893 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/0551857176174541961a8162c098cfd0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/0551857176174541961a8162c098cfd0 2023-04-16 20:53:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:50,893 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:50,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:50,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:50,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:50,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:50,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:50,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,898 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,898 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:50,898 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:50,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:50,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:50,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:50,898 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:50,898 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:50,904 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into 0551857176174541961a8162c098cfd0(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:50,904 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:50,904 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=12, startTime=1681678430333; duration=0sec 2023-04-16 20:53:50,904 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:50,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678490970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678490974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678490974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678490976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:50,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678490976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:50,998 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:50,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:51,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:51,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:51,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:51,008 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:51,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:51,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:51,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:51,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:51,013 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:51,013 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:51,014 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,014 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:51,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:51,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:51,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:51,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:51,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:51,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:51,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:51,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:51,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:51,028 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:51,028 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:51,029 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:51,029 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:51,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:51,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:51,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:51,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:51,030 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,031 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:51,031 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:51,031 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30eb24d0[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:51,033 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:51,033 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:51,034 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:51,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:51,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:51,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:51,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:51,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:51,037 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:51,037 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:51,037 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:51,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:51,040 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:51,040 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:51,040 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:51,040 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:51,040 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:51,044 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,044 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,044 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:51,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:51,046 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:51,048 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,050 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,050 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:51,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:51,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:51,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:51,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:51,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:51,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:51,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:51,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:51,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:51,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:51,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:51,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,052 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:51,052 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:51,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/dbb9b82239cf4bd79c6ae8a1a8a8a05a 2023-04-16 20:53:51,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/8c3854c29d5c40b7a68ea6daa4f562de as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8c3854c29d5c40b7a68ea6daa4f562de 2023-04-16 20:53:51,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8c3854c29d5c40b7a68ea6daa4f562de, entries=150, sequenceid=474, filesize=11.9 K 2023-04-16 20:53:51,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/fc10ea6f413946e4bf15533fffde5e0d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fc10ea6f413946e4bf15533fffde5e0d 2023-04-16 20:53:51,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fc10ea6f413946e4bf15533fffde5e0d, entries=150, sequenceid=474, filesize=11.9 K 2023-04-16 20:53:51,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/dbb9b82239cf4bd79c6ae8a1a8a8a05a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dbb9b82239cf4bd79c6ae8a1a8a8a05a 2023-04-16 20:53:51,109 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dbb9b82239cf4bd79c6ae8a1a8a8a05a, entries=150, sequenceid=474, filesize=11.9 K 2023-04-16 20:53:51,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~163.27 KB/167184, currentSize=140.89 KB/144270 for bfe0529baac322a72dbf6ef88dd5ff1c in 520ms, sequenceid=474, compaction requested=true 2023-04-16 20:53:51,110 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:51,110 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:51,110 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:51,110 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:51,110 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:51,110 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:51,111 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:51,112 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:51,112 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:51,112 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:51,112 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:51,112 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:51,112 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fb39135d43c74befb41bf8ddf67ab2d6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/db57cd3498b3406dbed04fc2a466fd1e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8c3854c29d5c40b7a68ea6daa4f562de] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.3 K 2023-04-16 20:53:51,112 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a8d8f6fca5e6471ebc5e5083b75750f9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1afc3e0b355d4cbfae26175cfadf2ee2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fc10ea6f413946e4bf15533fffde5e0d] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.3 K 2023-04-16 20:53:51,112 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting fb39135d43c74befb41bf8ddf67ab2d6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681678429657 2023-04-16 20:53:51,113 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting a8d8f6fca5e6471ebc5e5083b75750f9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681678429657 2023-04-16 20:53:51,113 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting db57cd3498b3406dbed04fc2a466fd1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681678430076 2023-04-16 20:53:51,114 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1afc3e0b355d4cbfae26175cfadf2ee2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681678430076 2023-04-16 20:53:51,114 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8c3854c29d5c40b7a68ea6daa4f562de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681678430583 2023-04-16 20:53:51,114 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting fc10ea6f413946e4bf15533fffde5e0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681678430583 2023-04-16 20:53:51,128 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:51,129 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:51,143 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/05144ed31f864e2ab3e1eaac30531976 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/05144ed31f864e2ab3e1eaac30531976 2023-04-16 20:53:51,147 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/7a37f10458e94127ae541d6dd24aeb95 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7a37f10458e94127ae541d6dd24aeb95 2023-04-16 20:53:51,152 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:51,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:51,156 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into 05144ed31f864e2ab3e1eaac30531976(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:51,156 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:51,156 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=13, startTime=1681678431110; duration=0sec 2023-04-16 20:53:51,156 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:51,157 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into 7a37f10458e94127ae541d6dd24aeb95(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:51,157 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:51,157 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:51,157 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=13, startTime=1681678431110; duration=0sec 2023-04-16 20:53:51,157 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:51,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:51,158 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:51,158 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:51,158 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:51,158 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/0551857176174541961a8162c098cfd0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3033d918c6d84c3ab7e7ab635206f628, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dbb9b82239cf4bd79c6ae8a1a8a8a05a] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.3 K 2023-04-16 20:53:51,159 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 0551857176174541961a8162c098cfd0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681678429657 2023-04-16 20:53:51,159 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3033d918c6d84c3ab7e7ab635206f628, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681678430076 2023-04-16 20:53:51,159 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting dbb9b82239cf4bd79c6ae8a1a8a8a05a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681678430583 2023-04-16 20:53:51,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:51,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:51,164 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:51,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:51,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:51,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:51,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:51,166 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:51,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:51,166 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:51,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:51,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:51,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:51,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:51,168 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:51,168 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:51,168 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:51,169 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:51,169 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:51,169 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:51,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:51,170 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=140.89 KB heapSize=381.47 KB 2023-04-16 20:53:51,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:51,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:51,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:51,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:51,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:51,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:51,184 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/ffe8ae1ec8df462d89cd5c2cb39934bb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ffe8ae1ec8df462d89cd5c2cb39934bb 2023-04-16 20:53:51,189 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into ffe8ae1ec8df462d89cd5c2cb39934bb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:51,189 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:51,189 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=13, startTime=1681678431110; duration=0sec 2023-04-16 20:53:51,190 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:51,275 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:51,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:51,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678491285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678491286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678491287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678491289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678491291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678491391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678491392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678491392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678491397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678491399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678491596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678491603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678491603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678491604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678491604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,607 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/812f1ef2522048b1b021b04bf9a59bfb 2023-04-16 20:53:51,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678491901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678491906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678491906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678491908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:51,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:51,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678491908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,036 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/a6f81b2ad7cb47f280bf8138478b1731 2023-04-16 20:53:52,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:52,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678492407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678492412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678492412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:52,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678492413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:52,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678492414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,454 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/bc6eaacd80664343a3c76ebbfa23ceb5 2023-04-16 20:53:52,465 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/812f1ef2522048b1b021b04bf9a59bfb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/812f1ef2522048b1b021b04bf9a59bfb 2023-04-16 20:53:52,472 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/812f1ef2522048b1b021b04bf9a59bfb, entries=150, sequenceid=510, filesize=11.9 K 2023-04-16 20:53:52,475 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/a6f81b2ad7cb47f280bf8138478b1731 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a6f81b2ad7cb47f280bf8138478b1731 2023-04-16 20:53:52,488 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a6f81b2ad7cb47f280bf8138478b1731, entries=150, sequenceid=510, filesize=11.9 K 2023-04-16 20:53:52,490 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/bc6eaacd80664343a3c76ebbfa23ceb5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/bc6eaacd80664343a3c76ebbfa23ceb5 2023-04-16 20:53:52,505 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/bc6eaacd80664343a3c76ebbfa23ceb5, entries=150, sequenceid=510, filesize=11.9 K 2023-04-16 20:53:52,506 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~381.42 KB/390576, currentSize=53.67 KB/54960 for bfe0529baac322a72dbf6ef88dd5ff1c in 1336ms, sequenceid=510, compaction requested=false 2023-04-16 20:53:52,506 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:52,506 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:52,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:52,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:52,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:52,507 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:52,509 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,509 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:52,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:52,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:52,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:52,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b464b33[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:52,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:52,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:52,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:52,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:52,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:52,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:52,522 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:52,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:52,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:52,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:52,522 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:52,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:52,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:52,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:52,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:52,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:52,526 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:52,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:52,533 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:52,533 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:52,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:52,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:52,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,534 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:52,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:52,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:52,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:52,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:52,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:52,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:52,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:52,535 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:52,535 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:52,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:52,635 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:52,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:52,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:52,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:52,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:52,641 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:52,641 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:52,641 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:52,641 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:52,641 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:52,641 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:52,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,644 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:52,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:52,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:52,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:52,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:52,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:52,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:52,651 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:52,651 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:52,651 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:52,651 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:52,651 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:52,651 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:52,651 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:52,652 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:53:52,652 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:52,652 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:52,652 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:52,652 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:52,652 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:52,652 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:52,686 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/dc0957be84fe42a8acf17d406846a8ca 2023-04-16 20:53:53,112 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/fd91825e894b4c2d8f08d6284399f003 2023-04-16 20:53:53,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:53,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:53,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681678433433, max timestamp=1681678433443Num uniques 859; 2023-04-16 20:53:53,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:53:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681678433433, max timestamp=1681678433442Num uniques 859; 2023-04-16 20:53:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:53:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678433433, max timestamp=1681678433442Num uniques 859; 2023-04-16 20:53:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:53:53,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678493463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678493463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678493464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678493465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678493465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,526 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=522 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/4fe8177a2145441d95f867f3b452981f 2023-04-16 20:53:53,532 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/dc0957be84fe42a8acf17d406846a8ca as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/dc0957be84fe42a8acf17d406846a8ca 2023-04-16 20:53:53,538 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/dc0957be84fe42a8acf17d406846a8ca, entries=150, sequenceid=522, filesize=11.9 K 2023-04-16 20:53:53,540 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/fd91825e894b4c2d8f08d6284399f003 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fd91825e894b4c2d8f08d6284399f003 2023-04-16 20:53:53,553 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fd91825e894b4c2d8f08d6284399f003, entries=150, sequenceid=522, filesize=11.9 K 2023-04-16 20:53:53,554 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/4fe8177a2145441d95f867f3b452981f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4fe8177a2145441d95f867f3b452981f 2023-04-16 20:53:53,560 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4fe8177a2145441d95f867f3b452981f, entries=150, sequenceid=522, filesize=11.9 K 2023-04-16 20:53:53,562 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for bfe0529baac322a72dbf6ef88dd5ff1c in 911ms, sequenceid=522, compaction requested=true 2023-04-16 20:53:53,562 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:53,562 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:53,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,565 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,565 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:53,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f72490c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:53,567 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:53,567 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,569 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,569 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:53,569 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:53,569 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:53,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:53,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:53,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-16 20:53:53,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1681678433461, max timestamp=1681678433572Num uniques 1068; 2023-04-16 20:53:53,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48903, new segment data size=6960, suffix heap size=139040,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=152 2023-04-16 20:53:53,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7684, totalHeapSize=17712, min timestamp=1681678433461, max timestamp=1681678433572Num uniques 1068; 2023-04-16 20:53:53,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48904, new segment data size=7684, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-16 20:53:53,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678433461, max timestamp=1681678433571Num uniques 1068; 2023-04-16 20:53:53,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:53,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:53,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:53,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:53,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:53,576 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-16 20:53:53,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:53,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:53,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:53,581 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:53,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:53,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:53,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:53,581 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:53,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:53,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,588 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,590 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/083f85fa9cce4956b29d4226c88673f1 2023-04-16 20:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:53,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:53,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:53,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,601 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:53,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:53,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:53,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:53,602 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:53,602 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:53,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:53,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678433596, max timestamp=1681678433612Num uniques 859; 2023-04-16 20:53:53,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:53,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678433596, max timestamp=1681678433612Num uniques 859; 2023-04-16 20:53:53,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:53,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678433596, max timestamp=1681678433612Num uniques 859; 2023-04-16 20:53:53,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 20:53:53,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/48c92263b9cf4f829c7d22213869bc2d 2023-04-16 20:53:53,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1681678433627, max timestamp=1681678433638Num uniques 1018; 2023-04-16 20:53:53,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1681678433627, max timestamp=1681678433638Num uniques 1018; 2023-04-16 20:53:53,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1681678433627, max timestamp=1681678433638Num uniques 1009; 2023-04-16 20:53:53,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=21428, suffix heap size=132904,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=468 2023-04-16 20:53:53,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=21428, suffix heap size=133840,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=468 2023-04-16 20:53:53,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=21428, suffix heap size=133840,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=468 2023-04-16 20:53:53,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678493653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678493654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678493657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678493659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678493661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,702 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:53,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:53,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:53,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:53,706 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:53,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:53,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:53,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:53,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,708 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:53,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:53,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:53,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:53,710 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:53,710 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:53,710 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:53,710 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:53,713 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,713 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,713 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,713 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:53,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4616d5e3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:53,715 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:53,715 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,716 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:53,717 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,717 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:53,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:53,721 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:53,721 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:53,721 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:53,721 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:53,721 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:53,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:53,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:53,723 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,725 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:53,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:53,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:53,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:53,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:53,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:53,733 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:53,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:53,733 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:53,733 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:53,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678493762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678493762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678493763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678493763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678493766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,833 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:53,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:53,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:53,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:53,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:53,837 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:53,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:53,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:53,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:53,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:53,839 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:53,839 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,839 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:53,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:53,840 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:53,840 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:53,841 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:53,841 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:53,842 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,842 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f32b455[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:53,844 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:53,845 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,846 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:53,846 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:53,846 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:53,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:53,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,849 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,849 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:53,849 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:53,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:53,853 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:53,853 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:53,853 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:53,853 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:53,853 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:53,855 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,855 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,855 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:53,856 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,857 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,859 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,859 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,859 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,859 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,860 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,860 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,860 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:53,860 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:53,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:53,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:53,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:53,860 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:53,860 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:53,860 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:53,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:53,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:53,960 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:53,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:53,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:53,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678493966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678493966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678493966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678493967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:53,972 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:53,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:53,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:53,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:53,973 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,973 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:53,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678493969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:53,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:53,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:53,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:53,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:53,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:53,977 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,977 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:53,978 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:53,978 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:53,978 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:53,978 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:53,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:53,981 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16146d53[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:53,983 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:53,983 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:53,985 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:53,985 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:53,985 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:53,986 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,986 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:53,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:53,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:53,991 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:53,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:53,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:53,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:53,991 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:53,993 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,993 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:53,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,996 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:53,996 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:53,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:54,004 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:54,004 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:54,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:54,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:54,004 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:54,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:54,004 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:54,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:54,004 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:54,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=12.71 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/154095ff0f874b9098f40f3f63a13cea 2023-04-16 20:53:54,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/083f85fa9cce4956b29d4226c88673f1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/083f85fa9cce4956b29d4226c88673f1 2023-04-16 20:53:54,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/083f85fa9cce4956b29d4226c88673f1, entries=200, sequenceid=562, filesize=14.3 K 2023-04-16 20:53:54,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/48c92263b9cf4f829c7d22213869bc2d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/48c92263b9cf4f829c7d22213869bc2d 2023-04-16 20:53:54,105 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:54,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:54,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:54,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/48c92263b9cf4f829c7d22213869bc2d, entries=150, sequenceid=562, filesize=11.9 K 2023-04-16 20:53:54,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/154095ff0f874b9098f40f3f63a13cea as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/154095ff0f874b9098f40f3f63a13cea 2023-04-16 20:53:54,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:54,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:54,137 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:54,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:54,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:54,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:54,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/154095ff0f874b9098f40f3f63a13cea, entries=150, sequenceid=562, filesize=11.9 K 2023-04-16 20:53:54,139 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~39.55 KB/40497, heapSize ~103.56 KB/106048, currentSize=154.31 KB/158010 for bfe0529baac322a72dbf6ef88dd5ff1c in 567ms, sequenceid=562, compaction requested=true 2023-04-16 20:53:54,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:54,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:54,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,139 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:54,139 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:54,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:54,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:53:54,139 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:54,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:54,140 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:54,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:54,142 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:54,142 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:54,142 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:54,142 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:54,143 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7a37f10458e94127ae541d6dd24aeb95, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a6f81b2ad7cb47f280bf8138478b1731, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fd91825e894b4c2d8f08d6284399f003, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/48c92263b9cf4f829c7d22213869bc2d] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=48.3 K 2023-04-16 20:53:54,143 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51934 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:54,143 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:54,143 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:54,143 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/05144ed31f864e2ab3e1eaac30531976, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/812f1ef2522048b1b021b04bf9a59bfb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/dc0957be84fe42a8acf17d406846a8ca, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/083f85fa9cce4956b29d4226c88673f1] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=50.7 K 2023-04-16 20:53:54,144 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 05144ed31f864e2ab3e1eaac30531976, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681678430583 2023-04-16 20:53:54,144 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7a37f10458e94127ae541d6dd24aeb95, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681678430583 2023-04-16 20:53:54,145 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 812f1ef2522048b1b021b04bf9a59bfb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1681678430652 2023-04-16 20:53:54,145 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting dc0957be84fe42a8acf17d406846a8ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1681678431285 2023-04-16 20:53:54,147 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting a6f81b2ad7cb47f280bf8138478b1731, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1681678430652 2023-04-16 20:53:54,147 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 083f85fa9cce4956b29d4226c88673f1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681678433570 2023-04-16 20:53:54,148 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting fd91825e894b4c2d8f08d6284399f003, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1681678431285 2023-04-16 20:53:54,148 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 48c92263b9cf4f829c7d22213869bc2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681678433571 2023-04-16 20:53:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:54,156 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:54,156 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:54,156 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=154.31 KB heapSize=415.78 KB 2023-04-16 20:53:54,156 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:54,156 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:54,156 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:54,156 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:54,157 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:54,157 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:54,170 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:54,182 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#80 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:54,203 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/7dfe88fce0b44b22a873b48c62f82ae4 2023-04-16 20:53:54,228 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/9b67ee67aaaa4198bcee1f3200f71774 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9b67ee67aaaa4198bcee1f3200f71774 2023-04-16 20:53:54,233 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/9366a0f26ece47f58ec9252d2d8ca781 2023-04-16 20:53:54,236 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into 9b67ee67aaaa4198bcee1f3200f71774(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:54,236 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:54,236 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=12, startTime=1681678434139; duration=0sec 2023-04-16 20:53:54,237 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:54,237 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:54,243 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:54,244 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:54,244 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:54,244 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ffe8ae1ec8df462d89cd5c2cb39934bb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/bc6eaacd80664343a3c76ebbfa23ceb5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4fe8177a2145441d95f867f3b452981f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/154095ff0f874b9098f40f3f63a13cea] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=48.3 K 2023-04-16 20:53:54,245 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ffe8ae1ec8df462d89cd5c2cb39934bb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681678430583 2023-04-16 20:53:54,247 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting bc6eaacd80664343a3c76ebbfa23ceb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1681678430652 2023-04-16 20:53:54,247 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 4fe8177a2145441d95f867f3b452981f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=522, earliestPutTs=1681678431285 2023-04-16 20:53:54,248 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 154095ff0f874b9098f40f3f63a13cea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681678433571 2023-04-16 20:53:54,268 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#83 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:54,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:54,274 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:54,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678494275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678494276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678494277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678494277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678494277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678494379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678494380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678494380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678494382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678494585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678494585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678494585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678494586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,633 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/6dc435c11e514836864862b69a699e20 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/6dc435c11e514836864862b69a699e20 2023-04-16 20:53:54,638 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into 6dc435c11e514836864862b69a699e20(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:54,638 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:54,638 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=12, startTime=1681678434139; duration=0sec 2023-04-16 20:53:54,639 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:54,672 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/4ff35fe71ad74d079b0160df233adeae 2023-04-16 20:53:54,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/7dfe88fce0b44b22a873b48c62f82ae4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7dfe88fce0b44b22a873b48c62f82ae4 2023-04-16 20:53:54,682 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/3a0deae3620a41bdaacfc74f729e0855 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3a0deae3620a41bdaacfc74f729e0855 2023-04-16 20:53:54,683 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7dfe88fce0b44b22a873b48c62f82ae4, entries=150, sequenceid=613, filesize=11.9 K 2023-04-16 20:53:54,684 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/9366a0f26ece47f58ec9252d2d8ca781 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9366a0f26ece47f58ec9252d2d8ca781 2023-04-16 20:53:54,689 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into 3a0deae3620a41bdaacfc74f729e0855(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:54,689 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:54,689 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=12, startTime=1681678434140; duration=0sec 2023-04-16 20:53:54,689 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:54,690 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9366a0f26ece47f58ec9252d2d8ca781, entries=150, sequenceid=613, filesize=11.9 K 2023-04-16 20:53:54,691 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/4ff35fe71ad74d079b0160df233adeae as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4ff35fe71ad74d079b0160df233adeae 2023-04-16 20:53:54,697 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4ff35fe71ad74d079b0160df233adeae, entries=150, sequenceid=613, filesize=11.9 K 2023-04-16 20:53:54,698 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~415.73 KB/425712, currentSize=33.54 KB/34350 for bfe0529baac322a72dbf6ef88dd5ff1c in 542ms, sequenceid=613, compaction requested=false 2023-04-16 20:53:54,698 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:54,698 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:54,701 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b136cb4[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:54,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:54,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,706 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:54,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:54,709 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:54,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:54,716 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:54,716 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:54,716 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:54,716 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:54,716 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:54,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:54,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:54,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:54,719 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,721 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:54,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:54,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:54,724 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:54,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:54,724 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:54,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:54,724 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,724 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,725 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:54,725 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:54,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:54,782 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:53:54,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:54,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:54,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:54,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:54,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:54,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:54,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/f5d36474a7374f56ab84b7f5a3bd5bd1 2023-04-16 20:53:54,825 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:54,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:54,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:54,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:54,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:54,830 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:54,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:54,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:54,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:54,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:54,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:54,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:54,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:54,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:54,848 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:54,848 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:54,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:54,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:54,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:54,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:54,848 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:54,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678434828, max timestamp=1681678434847Num uniques 859; 2023-04-16 20:53:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:54,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:53:54,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678434828, max timestamp=1681678434847Num uniques 859; 2023-04-16 20:53:54,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678434828, max timestamp=1681678434847Num uniques 859; 2023-04-16 20:53:54,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:53:54,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:53:54,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,853 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:54,853 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,853 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b5a0df8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:54,855 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:54,855 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,856 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:54,856 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,856 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:54,858 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:54,865 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:54,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:54,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:54,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:54,865 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:54,868 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,868 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,868 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:54,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:54,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:54,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,870 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:54,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:54,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:54,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,881 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,881 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,881 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:54,881 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:54,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:54,881 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,881 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:54,881 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:54,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:54,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:54,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678494896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678494896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678494896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678494897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:54,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678494897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,981 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:54,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:54,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:54,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:54,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:54,986 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:54,986 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:54,986 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:54,986 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:54,987 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,987 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:54,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:54,988 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,989 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,989 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:54,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:54,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:54,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:54,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:54,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:54,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:54,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:54,991 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:54,991 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:54,991 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:54,991 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:54,993 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61f08ada[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:54,995 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:54,995 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,997 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:54,997 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:54,997 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:55,000 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678495004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678495004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678495004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678495004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:55,006 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:55,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:55,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:55,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:55,006 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:55,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678495004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,009 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:55,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:55,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,021 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,021 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,021 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,021 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,021 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:55,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:55,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:55,022 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:55,022 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:55,021 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:55,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:55,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:55,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:55,122 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:55,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:55,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:55,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:55,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:55,127 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:55,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:55,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:55,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:55,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:55,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:55,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:55,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54bb0d42[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:55,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:55,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:55,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:55,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:55,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:55,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:55,155 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:55,164 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:55,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:55,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:55,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:55,164 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:55,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,172 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:55,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,172 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,177 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:55,178 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:55,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:55,178 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:55,177 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:55,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:55,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:55,178 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,178 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,178 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,178 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,178 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,178 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:55,178 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:55,179 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:55,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:55,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678495206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678495206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678495207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678495207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678495209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/aeb1e1b564164d109de9d9ecfb2a6194 2023-04-16 20:53:55,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:55,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:55,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:55,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:55,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:55,282 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:55,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:55,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:55,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:55,283 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,283 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,284 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:55,284 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:55,284 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,284 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:55,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:55,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:55,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:55,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:55,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:55,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:55,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:55,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:55,285 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,286 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:55,286 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:55,286 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:55,287 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64fdbdad[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:55,289 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:55,289 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:55,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:55,293 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:55,295 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:55,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:55,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:55,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:55,295 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:55,297 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,297 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,297 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,297 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,299 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:55,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:55,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:55,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:55,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:55,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:55,301 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:55,301 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:55,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:55,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:55,401 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:55,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:55,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:55,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:55,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:55,405 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:55,405 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:55,405 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:55,405 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:55,406 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,406 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,438 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:55,438 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:55,438 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,438 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:55,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:55,447 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:55,448 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,448 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:55,448 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:55,448 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:55,484 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:55,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f434907[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:55,486 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:55,486 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678495510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678495510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678495511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678495513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:55,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678495513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,561 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:55,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:55,630 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:55,630 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,630 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,630 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:55,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:55,635 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:55,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:55,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:55,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:55,635 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:55,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/ca9c8c1f3f3042a18904bd14ac3b29c7 2023-04-16 20:53:55,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/f5d36474a7374f56ab84b7f5a3bd5bd1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f5d36474a7374f56ab84b7f5a3bd5bd1 2023-04-16 20:53:55,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f5d36474a7374f56ab84b7f5a3bd5bd1, entries=200, sequenceid=626, filesize=14.3 K 2023-04-16 20:53:55,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/aeb1e1b564164d109de9d9ecfb2a6194 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aeb1e1b564164d109de9d9ecfb2a6194 2023-04-16 20:53:55,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:55,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:55,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:55,685 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,687 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aeb1e1b564164d109de9d9ecfb2a6194, entries=150, sequenceid=626, filesize=11.9 K 2023-04-16 20:53:55,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/ca9c8c1f3f3042a18904bd14ac3b29c7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ca9c8c1f3f3042a18904bd14ac3b29c7 2023-04-16 20:53:55,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ca9c8c1f3f3042a18904bd14ac3b29c7, entries=150, sequenceid=626, filesize=11.9 K 2023-04-16 20:53:55,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for bfe0529baac322a72dbf6ef88dd5ff1c in 916ms, sequenceid=626, compaction requested=true 2023-04-16 20:53:55,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:55,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:55,698 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:55,698 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:55,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:55,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:55,699 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:55,699 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:55,699 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:55,699 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:55,700 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,700 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,700 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/6dc435c11e514836864862b69a699e20, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9366a0f26ece47f58ec9252d2d8ca781, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aeb1e1b564164d109de9d9ecfb2a6194] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.6 K 2023-04-16 20:53:55,700 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9b67ee67aaaa4198bcee1f3200f71774, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7dfe88fce0b44b22a873b48c62f82ae4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f5d36474a7374f56ab84b7f5a3bd5bd1] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=38.9 K 2023-04-16 20:53:55,700 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6dc435c11e514836864862b69a699e20, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681678433571 2023-04-16 20:53:55,701 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9b67ee67aaaa4198bcee1f3200f71774, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681678433571 2023-04-16 20:53:55,701 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9366a0f26ece47f58ec9252d2d8ca781, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1681678433638 2023-04-16 20:53:55,701 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7dfe88fce0b44b22a873b48c62f82ae4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1681678433638 2023-04-16 20:53:55,701 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting f5d36474a7374f56ab84b7f5a3bd5bd1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681678434275 2023-04-16 20:53:55,701 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting aeb1e1b564164d109de9d9ecfb2a6194, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681678434275 2023-04-16 20:53:55,721 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:55,722 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,722 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,722 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,722 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:55,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:55,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:55,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:55,722 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:55,724 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:55,722 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,724 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:55,724 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,724 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:55,724 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:55,724 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:55,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:55,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:55,756 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/d341604a80d34a8b934a2e536c0f3d3b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d341604a80d34a8b934a2e536c0f3d3b 2023-04-16 20:53:55,764 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into d341604a80d34a8b934a2e536c0f3d3b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:55,764 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:55,764 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=13, startTime=1681678435698; duration=0sec 2023-04-16 20:53:55,765 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:55,765 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:55,766 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:55,766 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:55,767 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,767 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3a0deae3620a41bdaacfc74f729e0855, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4ff35fe71ad74d079b0160df233adeae, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ca9c8c1f3f3042a18904bd14ac3b29c7] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.6 K 2023-04-16 20:53:55,767 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3a0deae3620a41bdaacfc74f729e0855, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681678433571 2023-04-16 20:53:55,768 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 4ff35fe71ad74d079b0160df233adeae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1681678433638 2023-04-16 20:53:55,768 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ca9c8c1f3f3042a18904bd14ac3b29c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681678434275 2023-04-16 20:53:55,776 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/865bd4dc82704a85a6956ed39a11d782 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/865bd4dc82704a85a6956ed39a11d782 2023-04-16 20:53:55,785 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#89 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:55,787 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into 865bd4dc82704a85a6956ed39a11d782(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:55,787 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:55,787 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=13, startTime=1681678435698; duration=0sec 2023-04-16 20:53:55,788 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:55,824 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:55,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:55,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:55,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:55,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:55,828 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:55,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:55,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:55,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:55,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,905 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:55,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:55,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:55,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:55,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:55,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:55,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:55,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:55,907 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:55,908 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:55,908 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-16 20:53:55,908 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:55,908 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:55,908 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:55,908 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:55,908 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:55,908 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:56,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:56,017 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:56,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678496018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678496019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678496020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678496021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678496021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678496122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678496122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678496126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678496126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,208 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/d44dc21298594b4b97ffca875f90e7ee as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d44dc21298594b4b97ffca875f90e7ee 2023-04-16 20:53:56,214 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into d44dc21298594b4b97ffca875f90e7ee(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:56,214 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:56,214 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=13, startTime=1681678435698; duration=0sec 2023-04-16 20:53:56,214 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:56,318 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/7e37a0579ba1480f95d2b7cb9c32301e 2023-04-16 20:53:56,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678496324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678496324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678496329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678496330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678496629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678496630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678496632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:56,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678496633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,763 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/9a20dc96ab334ce0b87c61be72cd6bcd 2023-04-16 20:53:56,776 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/18f909a3b5ef4e5b88d6ee81761e6d77 2023-04-16 20:53:56,781 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/7e37a0579ba1480f95d2b7cb9c32301e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7e37a0579ba1480f95d2b7cb9c32301e 2023-04-16 20:53:56,786 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7e37a0579ba1480f95d2b7cb9c32301e, entries=150, sequenceid=665, filesize=11.9 K 2023-04-16 20:53:56,787 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/9a20dc96ab334ce0b87c61be72cd6bcd as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9a20dc96ab334ce0b87c61be72cd6bcd 2023-04-16 20:53:56,792 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9a20dc96ab334ce0b87c61be72cd6bcd, entries=150, sequenceid=665, filesize=11.9 K 2023-04-16 20:53:56,793 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/18f909a3b5ef4e5b88d6ee81761e6d77 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/18f909a3b5ef4e5b88d6ee81761e6d77 2023-04-16 20:53:56,803 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/18f909a3b5ef4e5b88d6ee81761e6d77, entries=150, sequenceid=665, filesize=11.9 K 2023-04-16 20:53:56,804 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for bfe0529baac322a72dbf6ef88dd5ff1c in 896ms, sequenceid=665, compaction requested=false 2023-04-16 20:53:56,804 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:56,804 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:56,806 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,806 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:56,806 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:56,806 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:56,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:56,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a00a86f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:56,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:56,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:56,810 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:56,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:56,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:56,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:56,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:56,811 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,811 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:56,813 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:56,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:56,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:56,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:56,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:56,816 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:56,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:56,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:56,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:56,817 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:56,818 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:56,818 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:56,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:56,818 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:56,820 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:56,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:56,825 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:56,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:56,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:56,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:56,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:56,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:56,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,826 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:56,826 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:56,826 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:56,826 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:56,826 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:56,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:56,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:56,926 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:56,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:56,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:56,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:56,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:56,929 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:56,929 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:56,929 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:56,929 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:56,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:56,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:56,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:56,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:56,931 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:56,932 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:56,932 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:56,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:56,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:56,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:56,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:56,933 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:56,933 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:56,933 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:56,933 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:56,933 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:56,933 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:56,933 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:56,934 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-16 20:53:56,934 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:56,934 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:56,934 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:56,934 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:56,934 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:56,934 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:56,949 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/fcb8bcacb4cd44b3a77d349b26e753d2 2023-04-16 20:53:56,971 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/80f53b494fdc47a99847c41261dac8f5 2023-04-16 20:53:56,992 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/08a9048508cd43ae8c4e38ce9a4e3c3c 2023-04-16 20:53:56,999 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/fcb8bcacb4cd44b3a77d349b26e753d2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fcb8bcacb4cd44b3a77d349b26e753d2 2023-04-16 20:53:57,005 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fcb8bcacb4cd44b3a77d349b26e753d2, entries=150, sequenceid=675, filesize=11.9 K 2023-04-16 20:53:57,008 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/80f53b494fdc47a99847c41261dac8f5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/80f53b494fdc47a99847c41261dac8f5 2023-04-16 20:53:57,014 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/80f53b494fdc47a99847c41261dac8f5, entries=150, sequenceid=675, filesize=11.9 K 2023-04-16 20:53:57,016 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/08a9048508cd43ae8c4e38ce9a4e3c3c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/08a9048508cd43ae8c4e38ce9a4e3c3c 2023-04-16 20:53:57,022 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/08a9048508cd43ae8c4e38ce9a4e3c3c, entries=150, sequenceid=675, filesize=11.9 K 2023-04-16 20:53:57,023 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for bfe0529baac322a72dbf6ef88dd5ff1c in 90ms, sequenceid=675, compaction requested=true 2023-04-16 20:53:57,023 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:57,023 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:57,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:57,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:57,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:57,026 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:57,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d6570[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:57,029 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:57,029 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,031 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:57,031 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:57,031 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:57,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:57,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,033 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:57,033 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:57,033 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:57,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:57,037 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:57,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:57,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:57,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:57,037 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:57,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:57,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:53:57,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:57,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:57,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:57,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:57,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:57,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:57,044 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,044 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:57,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,045 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,046 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,049 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:57,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:57,058 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:57,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:57,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,058 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:57,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:57,058 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:57,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:57,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:57,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:57,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:57,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/16113580032c4ede9ac7b76bff6338c7 2023-04-16 20:53:57,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678437090, max timestamp=1681678437103Num uniques 859; 2023-04-16 20:53:57,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:57,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678437090, max timestamp=1681678437103Num uniques 859; 2023-04-16 20:53:57,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:57,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678437090, max timestamp=1681678437103Num uniques 859; 2023-04-16 20:53:57,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:53:57,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678497147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678497149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678497149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678497150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678497151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,158 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:57,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:57,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:57,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:57,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:57,172 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:57,172 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:57,172 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:57,172 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:57,173 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,173 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,174 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:57,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:57,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:57,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:57,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:57,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:57,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:57,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,189 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,189 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,189 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc4b0c3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:57,192 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:57,192 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:57,194 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,194 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:57,200 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:57,200 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:57,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:57,207 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:57,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:57,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:57,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:57,207 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:57,210 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,210 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:57,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:57,210 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:57,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:57,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:57,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:57,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:57,221 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:57,222 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:57,222 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:57,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:57,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678497252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678497253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678497254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678497254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678497255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,322 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:57,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:57,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:57,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:57,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:57,326 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:57,326 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:57,326 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:57,326 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:57,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,328 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:57,328 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:57,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:57,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:57,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:57,330 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,330 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:57,330 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:57,330 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:57,332 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3acb4d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:57,334 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:57,334 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,344 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:57,344 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:57,344 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:57,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:57,346 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:57,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:57,349 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:57,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:57,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:57,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:57,349 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:57,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:57,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,354 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,356 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:57,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:57,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:57,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:57,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:57,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:57,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:57,362 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:57,362 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:57,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678497454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678497455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678497456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678497456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,462 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:57,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:57,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:57,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:57,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:57,465 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:57,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:57,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:57,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:57,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678497464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,468 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,468 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,470 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:57,470 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:57,470 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,470 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:57,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:57,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:57,471 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:57,471 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:57,471 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:57,472 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,472 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:57,472 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:57,472 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:57,473 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,473 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@757647b5[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:57,475 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:57,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:57,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:57,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:57,477 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,477 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:57,478 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:57,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:57,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:57,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:57,481 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:57,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:57,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:57,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:57,481 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:57,484 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,484 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:57,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,484 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:57,484 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,488 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,488 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,488 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,488 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,488 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:57,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:57,488 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:57,488 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:57,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:57,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:57,488 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:57,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:57,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:57,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:57,488 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:57,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/857e7b764fe84c55974560bc9b8cfb22 2023-04-16 20:53:57,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/b1dcf555d605427d8f23fd314a17f5d9 2023-04-16 20:53:57,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/16113580032c4ede9ac7b76bff6338c7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/16113580032c4ede9ac7b76bff6338c7 2023-04-16 20:53:57,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/16113580032c4ede9ac7b76bff6338c7, entries=150, sequenceid=685, filesize=11.9 K 2023-04-16 20:53:57,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/857e7b764fe84c55974560bc9b8cfb22 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/857e7b764fe84c55974560bc9b8cfb22 2023-04-16 20:53:57,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/857e7b764fe84c55974560bc9b8cfb22, entries=150, sequenceid=685, filesize=11.9 K 2023-04-16 20:53:57,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/b1dcf555d605427d8f23fd314a17f5d9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/b1dcf555d605427d8f23fd314a17f5d9 2023-04-16 20:53:57,538 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/b1dcf555d605427d8f23fd314a17f5d9, entries=150, sequenceid=685, filesize=11.9 K 2023-04-16 20:53:57,539 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bfe0529baac322a72dbf6ef88dd5ff1c in 499ms, sequenceid=685, compaction requested=true 2023-04-16 20:53:57,539 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:57,539 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:57,539 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:57,539 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:57,539 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:57,539 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:57,541 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:57,541 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:57,541 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,541 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/865bd4dc82704a85a6956ed39a11d782, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7e37a0579ba1480f95d2b7cb9c32301e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fcb8bcacb4cd44b3a77d349b26e753d2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/16113580032c4ede9ac7b76bff6338c7] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=48.6 K 2023-04-16 20:53:57,541 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 865bd4dc82704a85a6956ed39a11d782, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681678434275 2023-04-16 20:53:57,542 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7e37a0579ba1480f95d2b7cb9c32301e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1681678434894 2023-04-16 20:53:57,542 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting fcb8bcacb4cd44b3a77d349b26e753d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681678436019 2023-04-16 20:53:57,543 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 16113580032c4ede9ac7b76bff6338c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1681678437032 2023-04-16 20:53:57,544 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:57,544 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:57,545 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,545 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d341604a80d34a8b934a2e536c0f3d3b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9a20dc96ab334ce0b87c61be72cd6bcd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/80f53b494fdc47a99847c41261dac8f5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/857e7b764fe84c55974560bc9b8cfb22] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=48.6 K 2023-04-16 20:53:57,545 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d341604a80d34a8b934a2e536c0f3d3b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681678434275 2023-04-16 20:53:57,546 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9a20dc96ab334ce0b87c61be72cd6bcd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1681678434894 2023-04-16 20:53:57,546 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 80f53b494fdc47a99847c41261dac8f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681678436019 2023-04-16 20:53:57,546 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 857e7b764fe84c55974560bc9b8cfb22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1681678437032 2023-04-16 20:53:57,553 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#99 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:57,557 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#100 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:57,589 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:57,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:57,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:57,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:57,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:57,593 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:57,593 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:57,593 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:57,593 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:57,594 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,594 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,595 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,595 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:57,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:57,595 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,595 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:57,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:57,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:57,597 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,597 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:57,597 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 20:53:57,597 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:57,598 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:57,598 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:57,598 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:57,598 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:57,598 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:57,616 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=722 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/da35140ec13d49cca58d4d8d9264e814 2023-04-16 20:53:57,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:57,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:57,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678497763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678497766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678497766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678497767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678497769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678497869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678497870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678497871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:57,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678497877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:57,968 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/1cca72c30c5941c0a04ff4982168ea5d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1cca72c30c5941c0a04ff4982168ea5d 2023-04-16 20:53:57,975 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into 1cca72c30c5941c0a04ff4982168ea5d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:57,975 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:57,975 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=12, startTime=1681678437539; duration=0sec 2023-04-16 20:53:57,975 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:57,975 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:53:57,976 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/69642633f7434f9c9f9f3dd159a56037 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/69642633f7434f9c9f9f3dd159a56037 2023-04-16 20:53:57,977 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:53:57,977 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:57,977 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:57,977 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d44dc21298594b4b97ffca875f90e7ee, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/18f909a3b5ef4e5b88d6ee81761e6d77, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/08a9048508cd43ae8c4e38ce9a4e3c3c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/b1dcf555d605427d8f23fd314a17f5d9] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=48.6 K 2023-04-16 20:53:57,978 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d44dc21298594b4b97ffca875f90e7ee, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681678434275 2023-04-16 20:53:57,978 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 18f909a3b5ef4e5b88d6ee81761e6d77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1681678434894 2023-04-16 20:53:57,979 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 08a9048508cd43ae8c4e38ce9a4e3c3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1681678436019 2023-04-16 20:53:57,980 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting b1dcf555d605427d8f23fd314a17f5d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1681678437032 2023-04-16 20:53:57,982 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into 69642633f7434f9c9f9f3dd159a56037(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:57,982 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:57,982 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=12, startTime=1681678437539; duration=0sec 2023-04-16 20:53:57,982 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:57,991 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#103 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:58,038 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=722 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/9bd4e55e17a64e8d8e9c7666b73492e5 2023-04-16 20:53:58,051 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=722 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/d6baa6dbf0af4383ba2e6111213cf9a4 2023-04-16 20:53:58,057 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/da35140ec13d49cca58d4d8d9264e814 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/da35140ec13d49cca58d4d8d9264e814 2023-04-16 20:53:58,062 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/da35140ec13d49cca58d4d8d9264e814, entries=150, sequenceid=722, filesize=11.9 K 2023-04-16 20:53:58,063 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/9bd4e55e17a64e8d8e9c7666b73492e5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9bd4e55e17a64e8d8e9c7666b73492e5 2023-04-16 20:53:58,068 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9bd4e55e17a64e8d8e9c7666b73492e5, entries=150, sequenceid=722, filesize=11.9 K 2023-04-16 20:53:58,068 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/d6baa6dbf0af4383ba2e6111213cf9a4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d6baa6dbf0af4383ba2e6111213cf9a4 2023-04-16 20:53:58,072 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d6baa6dbf0af4383ba2e6111213cf9a4, entries=150, sequenceid=722, filesize=11.9 K 2023-04-16 20:53:58,073 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for bfe0529baac322a72dbf6ef88dd5ff1c in 476ms, sequenceid=722, compaction requested=false 2023-04-16 20:53:58,073 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:58,073 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:58,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:58,075 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:53:58,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:58,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:58,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:58,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:58,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:58,076 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:58,079 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:58,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@186dca8e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,084 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:58,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:58,086 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:58,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:58,089 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:58,089 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:58,089 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:58,089 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:58,089 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:58,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,091 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:58,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:58,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:58,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,095 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:58,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,096 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:58,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678498096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678438089, max timestamp=1681678438099Num uniques 859; 2023-04-16 20:53:58,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:53:58,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678438089, max timestamp=1681678438099Num uniques 859; 2023-04-16 20:53:58,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:53:58,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678438089, max timestamp=1681678438099Num uniques 859; 2023-04-16 20:53:58,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:53:58,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678498119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678498120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678498120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,196 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:58,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:58,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:58,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:58,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:58,201 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:58,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:58,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:58,202 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,202 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,204 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,205 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678498205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:58,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:58,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:58,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:58,220 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,220 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:58,221 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:58,221 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:58,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:58,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:58,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678498222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678498222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@320533e8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:58,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:58,230 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,230 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:58,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678498227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:58,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:58,238 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:58,238 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:58,238 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:58,238 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:58,238 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:58,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,240 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,241 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,241 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,245 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:58,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:58,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,246 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:58,246 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:58,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678498275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,346 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:58,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:58,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:58,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:58,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:58,349 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:58,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:58,350 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:58,350 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,350 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,351 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,351 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,351 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,351 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:58,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:58,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:58,353 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,353 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:58,353 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:58,353 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:58,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e49e646[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:58,357 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:58,358 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,359 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,359 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:58,359 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:58,359 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:58,360 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,360 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:58,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:58,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:58,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:58,364 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:58,364 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:58,364 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:58,364 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:58,364 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:58,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,366 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,368 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,371 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:58,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:58,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:58,372 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:58,372 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:58,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,402 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/d343ada7218a4a518cc91d47ee41ea0c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d343ada7218a4a518cc91d47ee41ea0c 2023-04-16 20:53:58,407 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into d343ada7218a4a518cc91d47ee41ea0c(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:58,407 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:58,407 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=12, startTime=1681678437539; duration=0sec 2023-04-16 20:53:58,407 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:58,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678498409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678498427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678498429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678498432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,472 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:58,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:58,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:58,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:58,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:58,476 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:58,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:58,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:58,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,477 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,478 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,478 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:58,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:58,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:58,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:58,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:58,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:58,479 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,479 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:58,479 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:58,479 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:58,481 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,481 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,481 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@594551d5[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:58,483 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:58,483 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,484 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/ff4d0a124a23434aaeb3cbd28fc55ab9 2023-04-16 20:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:58,484 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:58,486 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:58,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:58,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:58,487 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:58,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:58,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:58,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:58,487 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:58,489 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,489 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,489 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,490 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,491 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:58,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:58,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:58,494 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:58,494 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:58,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,515 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/21724f73a39f40a3a4d0ab4fa63b10bb 2023-04-16 20:53:58,594 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:58,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:58,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:58,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:58,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:58,598 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:58,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:58,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:58,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,599 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,599 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:58,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:58,601 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:58,601 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,601 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:58,601 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:58,601 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:58,601 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,602 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:58,602 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:58,602 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:58,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:58,604 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,604 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:58,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8ac9c9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:58,608 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:58,608 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,609 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:58,612 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,612 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:58,613 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,613 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:58,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:58,615 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:58,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:58,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:58,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:58,615 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:58,616 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,617 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,618 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,627 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:58,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:58,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:58,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,628 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,628 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:58,628 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:58,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678498712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,729 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:58,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:58,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:58,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:58,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:58,732 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:58,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:58,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:58,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,733 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:58,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:58,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:58,735 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,736 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:58,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678498734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,736 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:58,736 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:58,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678498734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:58,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:58,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678498735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:58,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@348b25a3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:58,739 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:58,739 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,740 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:58,741 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,741 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:58,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:58,741 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:58,743 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:58,745 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:58,746 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:58,746 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:58,746 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:58,746 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:58,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,749 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,751 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,751 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,751 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:58,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:58,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:58,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,752 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,752 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,752 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,752 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,752 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:58,752 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:58,752 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,752 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,852 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:58,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:58,857 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:58,857 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,857 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:58,857 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:58,857 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,857 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,859 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,859 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,859 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,859 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:58,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:58,868 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,868 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:58,868 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:58,868 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:58,876 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,877 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:58,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65219ea2[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:58,879 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:58,879 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:58,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:58,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:58,880 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:58,881 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:58,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:58,884 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:58,884 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:58,884 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:58,884 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:58,884 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:58,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:58,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:58,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,886 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,889 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:58,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:58,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:58,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:58,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:58,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:58,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:58,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:58,893 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:58,894 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:58,894 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:58,894 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:58,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:58,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/2086b9c5aff647dd96b675b9acd9efb8 2023-04-16 20:53:58,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/ff4d0a124a23434aaeb3cbd28fc55ab9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ff4d0a124a23434aaeb3cbd28fc55ab9 2023-04-16 20:53:58,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ff4d0a124a23434aaeb3cbd28fc55ab9, entries=200, sequenceid=735, filesize=14.3 K 2023-04-16 20:53:58,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/21724f73a39f40a3a4d0ab4fa63b10bb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/21724f73a39f40a3a4d0ab4fa63b10bb 2023-04-16 20:53:58,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/21724f73a39f40a3a4d0ab4fa63b10bb, entries=150, sequenceid=735, filesize=11.9 K 2023-04-16 20:53:58,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/2086b9c5aff647dd96b675b9acd9efb8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2086b9c5aff647dd96b675b9acd9efb8 2023-04-16 20:53:58,956 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2086b9c5aff647dd96b675b9acd9efb8, entries=150, sequenceid=735, filesize=11.9 K 2023-04-16 20:53:58,957 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for bfe0529baac322a72dbf6ef88dd5ff1c in 882ms, sequenceid=735, compaction requested=true 2023-04-16 20:53:58,957 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:58,957 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:58,958 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:58,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:58,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:53:58,958 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:58,959 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:58,959 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:58,959 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:53:58,959 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:53:58,959 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,960 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:58,960 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1cca72c30c5941c0a04ff4982168ea5d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/da35140ec13d49cca58d4d8d9264e814, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ff4d0a124a23434aaeb3cbd28fc55ab9] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=39.2 K 2023-04-16 20:53:58,960 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/69642633f7434f9c9f9f3dd159a56037, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9bd4e55e17a64e8d8e9c7666b73492e5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/21724f73a39f40a3a4d0ab4fa63b10bb] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.8 K 2023-04-16 20:53:58,960 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1cca72c30c5941c0a04ff4982168ea5d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1681678437032 2023-04-16 20:53:58,960 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 69642633f7434f9c9f9f3dd159a56037, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1681678437032 2023-04-16 20:53:58,960 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting da35140ec13d49cca58d4d8d9264e814, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1681678437147 2023-04-16 20:53:58,961 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9bd4e55e17a64e8d8e9c7666b73492e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1681678437147 2023-04-16 20:53:58,961 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ff4d0a124a23434aaeb3cbd28fc55ab9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681678437762 2023-04-16 20:53:58,961 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 21724f73a39f40a3a4d0ab4fa63b10bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681678437765 2023-04-16 20:53:58,976 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#109 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:58,976 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#108 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:58,994 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:58,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:58,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:58,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:58,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:58,998 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:58,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:58,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:58,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:58,999 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:58,999 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,000 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,000 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:59,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:59,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:59,003 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-16 20:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:59,004 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:59,004 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:53:59,006 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/c2317acf57434a05acb3e72ec6e1c53a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c2317acf57434a05acb3e72ec6e1c53a 2023-04-16 20:53:59,035 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into c2317acf57434a05acb3e72ec6e1c53a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:59,035 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:59,035 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=13, startTime=1681678438957; duration=0sec 2023-04-16 20:53:59,035 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:53:59,035 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:53:59,037 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:53:59,037 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:53:59,037 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,037 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d343ada7218a4a518cc91d47ee41ea0c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d6baa6dbf0af4383ba2e6111213cf9a4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2086b9c5aff647dd96b675b9acd9efb8] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.8 K 2023-04-16 20:53:59,038 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d343ada7218a4a518cc91d47ee41ea0c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1681678437032 2023-04-16 20:53:59,038 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d6baa6dbf0af4383ba2e6111213cf9a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=722, earliestPutTs=1681678437147 2023-04-16 20:53:59,038 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2086b9c5aff647dd96b675b9acd9efb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681678437765 2023-04-16 20:53:59,043 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/393fe2d0674d4c1089815a1aa72a5ab8 2023-04-16 20:53:59,046 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#111 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:53:59,078 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/01a1bf4071f645cf859cc74944c73ac6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/01a1bf4071f645cf859cc74944c73ac6 2023-04-16 20:53:59,078 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/1a4724f5e49e4eb2bd4004ed55eb4c9f 2023-04-16 20:53:59,094 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into 01a1bf4071f645cf859cc74944c73ac6(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:59,094 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:59,094 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=13, startTime=1681678438958; duration=0sec 2023-04-16 20:53:59,094 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:59,116 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/a7b6912feb6f41488f479d0ffe1ad5be 2023-04-16 20:53:59,124 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/393fe2d0674d4c1089815a1aa72a5ab8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/393fe2d0674d4c1089815a1aa72a5ab8 2023-04-16 20:53:59,131 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/393fe2d0674d4c1089815a1aa72a5ab8, entries=150, sequenceid=771, filesize=11.9 K 2023-04-16 20:53:59,132 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/1a4724f5e49e4eb2bd4004ed55eb4c9f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1a4724f5e49e4eb2bd4004ed55eb4c9f 2023-04-16 20:53:59,148 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1a4724f5e49e4eb2bd4004ed55eb4c9f, entries=150, sequenceid=771, filesize=11.9 K 2023-04-16 20:53:59,150 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/a7b6912feb6f41488f479d0ffe1ad5be as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a7b6912feb6f41488f479d0ffe1ad5be 2023-04-16 20:53:59,157 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a7b6912feb6f41488f479d0ffe1ad5be, entries=150, sequenceid=771, filesize=11.9 K 2023-04-16 20:53:59,158 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for bfe0529baac322a72dbf6ef88dd5ff1c in 155ms, sequenceid=771, compaction requested=false 2023-04-16 20:53:59,158 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:59,158 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,160 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:59,160 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:59,161 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,161 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,162 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,162 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1619a02a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:59,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:59,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,165 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,166 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:59,166 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:59,166 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:59,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:59,170 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:59,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:59,175 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:59,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:59,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:59,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:59,175 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:59,177 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,177 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,177 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,180 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,185 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,185 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,185 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:59,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:59,185 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,185 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,185 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,185 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,186 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:59,186 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:59,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:53:59,227 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:53:59,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:53:59,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:59,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:53:59,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:59,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:53:59,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:53:59,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681678439242, max timestamp=1681678439251Num uniques 859; 2023-04-16 20:53:59,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-16 20:53:59,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681678439242, max timestamp=1681678439251Num uniques 859; 2023-04-16 20:53:59,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-16 20:53:59,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681678439242, max timestamp=1681678439251Num uniques 859; 2023-04-16 20:53:59,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-16 20:53:59,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/9ed8b43249be4e42893b8c0b265fb465 2023-04-16 20:53:59,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678499271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678499272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678499272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678499274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,286 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:59,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:59,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:59,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678499289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:59,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:59,296 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:59,296 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,296 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:59,296 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:59,297 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,297 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,299 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,299 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:59,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:59,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:59,315 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:59,315 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,315 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:59,316 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,316 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:59,316 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:59,316 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:59,316 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:59,318 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,318 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa4f7a1[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:59,321 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:59,321 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:59,322 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,322 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:59,323 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,324 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:59,324 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,324 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:59,327 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:59,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:59,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:59,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:59,327 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:59,330 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,330 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,330 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,331 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:59,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:59,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,336 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:59,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,336 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:59,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,337 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678499379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678499381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678499383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678499383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,412 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/7886bb9feb53469ab7ffecfb4c81e132 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7886bb9feb53469ab7ffecfb4c81e132 2023-04-16 20:53:59,421 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into 7886bb9feb53469ab7ffecfb4c81e132(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:53:59,421 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:53:59,421 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=13, startTime=1681678438958; duration=0sec 2023-04-16 20:53:59,421 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:53:59,436 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:59,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:59,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:59,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:59,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:59,440 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:59,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:59,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:59,441 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,441 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,442 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,442 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,442 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,442 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:59,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:59,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:59,444 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,444 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:59,444 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:59,444 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:59,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:59,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ef71cb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:59,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:59,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,453 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,453 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:59,453 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:59,453 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,453 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:59,453 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:59,455 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:59,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:59,458 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:59,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:59,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:59,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:59,458 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:59,460 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,460 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,460 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,461 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,462 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:59,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:59,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:59,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,465 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:59,465 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:59,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,565 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:59,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:59,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:59,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:59,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:59,569 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:59,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:59,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:59,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,571 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,572 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,572 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:59,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:59,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:59,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:59,573 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:59,574 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:59,574 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:59,574 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:59,574 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:59,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:59,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:59,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:59,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,575 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:59,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b522bba[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:59,577 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:59,577 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,578 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:59,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:59,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:59,579 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,579 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:59,580 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:59,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:59,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:59,584 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:59,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:59,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:59,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:59,584 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:59,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678499585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678499585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678499585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,589 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678499587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,597 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:59,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:59,597 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:59,597 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:59,698 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:59,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:59,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:59,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:59,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:59,702 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:59,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:59,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:59,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,706 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/3cb81e5695784110a1847549dd84cb30 2023-04-16 20:53:59,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:59,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:59,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:59,711 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:59,711 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,711 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:59,724 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,724 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:59,724 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:59,724 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:59,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:59,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3011ce0a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:59,730 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:59,731 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,732 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:59,732 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:59,732 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:59,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:59,733 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:59,733 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,733 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,733 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:59,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:59,737 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:59,737 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:59,737 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:59,737 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:59,737 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:59,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,741 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,743 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:59,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:59,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,748 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:59,748 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:59,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,848 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:59,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:59,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:59,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:59,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:59,852 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:59,852 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,852 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:59,852 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:59,852 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,852 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,853 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,854 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,854 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:59,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:59,855 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,855 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:59,855 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:59,855 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:59,857 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:59,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c02e56[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:59,859 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:59,859 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,860 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:59,860 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,860 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:59,863 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,863 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,863 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,863 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:59,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:59,865 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:59,866 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:59,866 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:59,866 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:59,866 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:59,867 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,867 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,867 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,868 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,871 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:59,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:59,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,872 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,872 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:59,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:53:59,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:53:59,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678499890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678499890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678499890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:53:59,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678499891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:53:59,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:53:59,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:53:59,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:53:59,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:53:59,975 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:53:59,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:53:59,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:53:59,976 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,976 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,977 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,977 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,977 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,977 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:53:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:53:59,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:53:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:53:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:53:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:53:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:53:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:53:59,979 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,979 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:53:59,979 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:53:59,979 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:53:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:53:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:53:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:53:59,980 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:53:59,982 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:53:59,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d879ac[Count = 0] remaining members to acquire global barrier 2023-04-16 20:53:59,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:53:59,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,985 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,985 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:53:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:53:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:53:59,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:53:59,989 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:53:59,989 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,989 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,989 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:53:59,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:53:59,991 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:53:59,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:53:59,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:53:59,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:53:59,991 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:53:59,993 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,993 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,993 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:53:59,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:53:59,994 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,995 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,998 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:53:59,998 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:53:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:53:59,998 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:53:59,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:53:59,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:53:59,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:53:59,998 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,998 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:53:59,998 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:53:59,998 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,998 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:53:59,998 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:53:59,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:53:59,998 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:53:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:53:59,998 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:00,098 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:00,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:00,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:00,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:00,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:00,102 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:00,103 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,103 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:00,103 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:00,103 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,103 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:00,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:00,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:00,114 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,114 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:54:00,114 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:54:00,114 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,117 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,117 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b2d9dd6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:00,120 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:00,120 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,125 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,125 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,125 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:00,125 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:00,125 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:00,125 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:00,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:00,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,127 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:00,130 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:00,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:00,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:00,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:00,130 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:00,134 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,134 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,134 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,134 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:00,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:00,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,139 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,139 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,139 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,139 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:00,139 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:00,139 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/7006bae1e56546da929ecdcff12b547d 2023-04-16 20:54:00,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/9ed8b43249be4e42893b8c0b265fb465 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9ed8b43249be4e42893b8c0b265fb465 2023-04-16 20:54:00,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9ed8b43249be4e42893b8c0b265fb465, entries=150, sequenceid=783, filesize=11.9 K 2023-04-16 20:54:00,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/3cb81e5695784110a1847549dd84cb30 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3cb81e5695784110a1847549dd84cb30 2023-04-16 20:54:00,166 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3cb81e5695784110a1847549dd84cb30, entries=150, sequenceid=783, filesize=11.9 K 2023-04-16 20:54:00,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/7006bae1e56546da929ecdcff12b547d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7006bae1e56546da929ecdcff12b547d 2023-04-16 20:54:00,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7006bae1e56546da929ecdcff12b547d, entries=150, sequenceid=783, filesize=11.9 K 2023-04-16 20:54:00,172 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for bfe0529baac322a72dbf6ef88dd5ff1c in 945ms, sequenceid=783, compaction requested=true 2023-04-16 20:54:00,172 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:00,172 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:00,172 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:00,172 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:00,172 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:00,172 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:00,173 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:00,173 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:00,173 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:54:00,173 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:54:00,173 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,174 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,174 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7886bb9feb53469ab7ffecfb4c81e132, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1a4724f5e49e4eb2bd4004ed55eb4c9f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3cb81e5695784110a1847549dd84cb30] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.9 K 2023-04-16 20:54:00,174 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c2317acf57434a05acb3e72ec6e1c53a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/393fe2d0674d4c1089815a1aa72a5ab8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9ed8b43249be4e42893b8c0b265fb465] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.9 K 2023-04-16 20:54:00,174 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7886bb9feb53469ab7ffecfb4c81e132, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681678437765 2023-04-16 20:54:00,174 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c2317acf57434a05acb3e72ec6e1c53a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681678437765 2023-04-16 20:54:00,174 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1a4724f5e49e4eb2bd4004ed55eb4c9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681678438114 2023-04-16 20:54:00,174 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 393fe2d0674d4c1089815a1aa72a5ab8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681678438114 2023-04-16 20:54:00,175 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3cb81e5695784110a1847549dd84cb30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681678439222 2023-04-16 20:54:00,175 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9ed8b43249be4e42893b8c0b265fb465, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681678439222 2023-04-16 20:54:00,181 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#118 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:00,181 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#117 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:00,200 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/f8e3cf7c04b542ecbc0b6d133d3e2a92 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f8e3cf7c04b542ecbc0b6d133d3e2a92 2023-04-16 20:54:00,200 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/d340b7416a6c4247b1d2d55e1dfadb3b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d340b7416a6c4247b1d2d55e1dfadb3b 2023-04-16 20:54:00,206 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into d340b7416a6c4247b1d2d55e1dfadb3b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:00,206 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:00,206 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=13, startTime=1681678440172; duration=0sec 2023-04-16 20:54:00,206 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:00,206 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:00,207 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into f8e3cf7c04b542ecbc0b6d133d3e2a92(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:00,207 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:00,207 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=13, startTime=1681678440172; duration=0sec 2023-04-16 20:54:00,207 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:00,207 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:00,207 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:54:00,208 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,208 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/01a1bf4071f645cf859cc74944c73ac6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a7b6912feb6f41488f479d0ffe1ad5be, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7006bae1e56546da929ecdcff12b547d] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=36.9 K 2023-04-16 20:54:00,208 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 01a1bf4071f645cf859cc74944c73ac6, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681678437765 2023-04-16 20:54:00,209 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting a7b6912feb6f41488f479d0ffe1ad5be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681678438114 2023-04-16 20:54:00,209 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7006bae1e56546da929ecdcff12b547d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681678439222 2023-04-16 20:54:00,216 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#119 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:00,239 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:00,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:00,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:00,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:00,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:00,243 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:00,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:00,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:00,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,247 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,247 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:00,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:00,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:00,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:00,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:00,255 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,255 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:54:00,255 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-04-16 20:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:00,305 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=821 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/8099355338ce4dd193bda154583cd895 2023-04-16 20:54:00,349 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=821 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/3dd16b462bd6483897f15ece881f19b6 2023-04-16 20:54:00,371 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=821 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/aadab7472118421ea617ce4fe62d5422 2023-04-16 20:54:00,384 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/8099355338ce4dd193bda154583cd895 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8099355338ce4dd193bda154583cd895 2023-04-16 20:54:00,391 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8099355338ce4dd193bda154583cd895, entries=150, sequenceid=821, filesize=11.9 K 2023-04-16 20:54:00,392 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/3dd16b462bd6483897f15ece881f19b6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3dd16b462bd6483897f15ece881f19b6 2023-04-16 20:54:00,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:54:00,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:54:00,405 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3dd16b462bd6483897f15ece881f19b6, entries=150, sequenceid=821, filesize=11.9 K 2023-04-16 20:54:00,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678500403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,406 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/aadab7472118421ea617ce4fe62d5422 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/aadab7472118421ea617ce4fe62d5422 2023-04-16 20:54:00,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678500404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678500404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678500405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,412 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/aadab7472118421ea617ce4fe62d5422, entries=150, sequenceid=821, filesize=11.9 K 2023-04-16 20:54:00,413 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=40.25 KB/41220 for bfe0529baac322a72dbf6ef88dd5ff1c in 158ms, sequenceid=821, compaction requested=false 2023-04-16 20:54:00,413 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:00,413 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,414 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:00,414 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,416 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45f26d27[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:00,418 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:00,418 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,420 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:00,420 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,420 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:00,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:00,424 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:00,425 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:00,425 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:00,425 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:00,425 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:00,428 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,428 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,429 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,433 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,436 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,436 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,436 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:00,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:00,437 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:00,437 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:00,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:00,437 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,437 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,437 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,437 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,437 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,437 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,437 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:54:00,509 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:54:00,509 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:54:00,509 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:00,509 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:54:00,509 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:00,509 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:54:00,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:00,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678440522, max timestamp=1681678440527Num uniques 859; 2023-04-16 20:54:00,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678440522, max timestamp=1681678440527Num uniques 859; 2023-04-16 20:54:00,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678440522, max timestamp=1681678440527Num uniques 859; 2023-04-16 20:54:00,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:00,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:00,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:00,537 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:00,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:00,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:00,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:00,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:00,540 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:00,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:00,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:00,541 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,541 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,542 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,542 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,542 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,543 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:00,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:00,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:00,547 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:00,547 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:00,547 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,547 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:54:00,547 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:54:00,548 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:00,549 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:00,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb95951[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:00,553 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:00,553 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678500556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:00,560 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,560 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:00,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678500558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678500559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678500559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:00,569 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:00,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:00,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:00,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:00,569 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:00,570 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,570 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,570 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,571 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,573 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,574 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:00,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:00,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,575 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:00,575 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:00,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,627 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/f9ae11b0df5b4d57a587cbccde3c6384 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f9ae11b0df5b4d57a587cbccde3c6384 2023-04-16 20:54:00,632 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into f9ae11b0df5b4d57a587cbccde3c6384(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:00,632 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:00,632 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=13, startTime=1681678440172; duration=0sec 2023-04-16 20:54:00,632 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:00,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678500661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678500664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678500665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678500665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,675 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:00,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:00,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:00,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:00,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:00,680 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:00,680 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,680 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:00,680 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:00,682 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,682 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,684 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,684 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,685 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:00,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:00,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:00,689 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:00,690 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,690 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:54:00,690 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:54:00,690 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:00,691 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:00,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eee5273[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:00,694 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:00,694 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:00,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:00,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:00,695 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,695 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,701 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:00,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:00,704 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:00,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:00,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:00,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:00,704 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:00,705 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,705 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,706 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:00,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:00,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,711 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,711 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:00,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:00,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,811 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:00,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:00,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:00,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:00,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:00,815 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:00,815 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,815 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:00,816 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:00,816 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,816 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,818 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,818 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,818 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,818 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:00,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:00,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:00,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:00,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:00,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:00,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:00,838 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:00,838 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:00,838 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,838 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. started... 2023-04-16 20:54:00,838 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. as already flushing 2023-04-16 20:54:00,838 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:00,838 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:00,838 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:00,838 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,839 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:00,839 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:00,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,841 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@693ab355[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:00,843 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:00,843 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,844 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,845 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:00,845 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:00,845 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:00,845 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,845 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:00,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,847 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,847 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,847 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:00,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:00,850 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:00,851 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:00,851 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:00,851 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:00,851 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:00,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,854 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:00,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,856 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678500865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678500866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:00,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678500867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678500867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:00,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:00,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:00,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:00,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:00,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:00,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:00,872 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,872 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:00,872 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:00,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:00,872 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:00,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:00,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:00,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/5b1bf1200c754ebbb84604fabb615b62 2023-04-16 20:54:00,968 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b02eeb3 to 127.0.0.1:54578 2023-04-16 20:54:00,968 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:00,968 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1818ad0a to 127.0.0.1:54578 2023-04-16 20:54:00,968 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:00,968 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:54578 2023-04-16 20:54:00,969 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:00,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:00,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:00,973 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6129209e to 127.0.0.1:54578 2023-04-16 20:54:00,973 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:01,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:01,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:01,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678501169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:01,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678501169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:01,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:01,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:01,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678501173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:01,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678501173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:01,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:01,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43186 deadline: 1681678501303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:01,305 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:01,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/0ec53684f9004d3abc8f95e63a57d37d 2023-04-16 20:54:01,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:01,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43196 deadline: 1681678501670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:01,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:01,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43174 deadline: 1681678501671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:01,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:01,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43226 deadline: 1681678501676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:01,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:01,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43166 deadline: 1681678501676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:01,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/f35de83c862b47699070e8deac8af058 2023-04-16 20:54:01,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/5b1bf1200c754ebbb84604fabb615b62 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5b1bf1200c754ebbb84604fabb615b62 2023-04-16 20:54:01,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5b1bf1200c754ebbb84604fabb615b62, entries=200, sequenceid=831, filesize=14.3 K 2023-04-16 20:54:01,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/0ec53684f9004d3abc8f95e63a57d37d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/0ec53684f9004d3abc8f95e63a57d37d 2023-04-16 20:54:01,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/0ec53684f9004d3abc8f95e63a57d37d, entries=150, sequenceid=831, filesize=11.9 K 2023-04-16 20:54:01,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/f35de83c862b47699070e8deac8af058 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f35de83c862b47699070e8deac8af058 2023-04-16 20:54:01,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f35de83c862b47699070e8deac8af058, entries=150, sequenceid=831, filesize=11.9 K 2023-04-16 20:54:01,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bfe0529baac322a72dbf6ef88dd5ff1c in 1260ms, sequenceid=831, compaction requested=true 2023-04-16 20:54:01,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:01,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:01,769 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:01,769 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:01,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:01,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:01,770 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40326 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:01,770 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/A is initiating minor compaction (all files) 2023-04-16 20:54:01,770 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:01,770 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/A in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:01,770 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/B is initiating minor compaction (all files) 2023-04-16 20:54:01,770 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f8e3cf7c04b542ecbc0b6d133d3e2a92, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8099355338ce4dd193bda154583cd895, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5b1bf1200c754ebbb84604fabb615b62] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=39.4 K 2023-04-16 20:54:01,770 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/B in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:01,770 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d340b7416a6c4247b1d2d55e1dfadb3b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3dd16b462bd6483897f15ece881f19b6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/0ec53684f9004d3abc8f95e63a57d37d] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=37.0 K 2023-04-16 20:54:01,771 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting f8e3cf7c04b542ecbc0b6d133d3e2a92, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681678439222 2023-04-16 20:54:01,771 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d340b7416a6c4247b1d2d55e1dfadb3b, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681678439222 2023-04-16 20:54:01,771 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8099355338ce4dd193bda154583cd895, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=821, earliestPutTs=1681678439268 2023-04-16 20:54:01,771 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3dd16b462bd6483897f15ece881f19b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=821, earliestPutTs=1681678439268 2023-04-16 20:54:01,772 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 5b1bf1200c754ebbb84604fabb615b62, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1681678440397 2023-04-16 20:54:01,772 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0ec53684f9004d3abc8f95e63a57d37d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1681678440397 2023-04-16 20:54:01,779 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#A#compaction#126 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:01,780 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#B#compaction#127 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:02,189 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/3dddaf41997f46ce96dc50a22df65819 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/3dddaf41997f46ce96dc50a22df65819 2023-04-16 20:54:02,189 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/7d3073f046284495ae61878068072fad as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7d3073f046284495ae61878068072fad 2023-04-16 20:54:02,194 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/B of bfe0529baac322a72dbf6ef88dd5ff1c into 7d3073f046284495ae61878068072fad(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:02,194 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:02,194 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/B, priority=13, startTime=1681678441769; duration=0sec 2023-04-16 20:54:02,195 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/A of bfe0529baac322a72dbf6ef88dd5ff1c into 3dddaf41997f46ce96dc50a22df65819(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:02,195 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:02,195 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:02,195 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:02,195 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/A, priority=13, startTime=1681678441769; duration=0sec 2023-04-16 20:54:02,195 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:02,196 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:02,196 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): bfe0529baac322a72dbf6ef88dd5ff1c/C is initiating minor compaction (all files) 2023-04-16 20:54:02,196 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfe0529baac322a72dbf6ef88dd5ff1c/C in TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:02,196 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f9ae11b0df5b4d57a587cbccde3c6384, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/aadab7472118421ea617ce4fe62d5422, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f35de83c862b47699070e8deac8af058] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp, totalSize=37.0 K 2023-04-16 20:54:02,196 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting f9ae11b0df5b4d57a587cbccde3c6384, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681678439222 2023-04-16 20:54:02,197 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting aadab7472118421ea617ce4fe62d5422, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=821, earliestPutTs=1681678439268 2023-04-16 20:54:02,197 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting f35de83c862b47699070e8deac8af058, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1681678440397 2023-04-16 20:54:02,204 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): bfe0529baac322a72dbf6ef88dd5ff1c#C#compaction#128 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:02,613 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/d1e1a4cea29247ca9bb53043ff867f83 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d1e1a4cea29247ca9bb53043ff867f83 2023-04-16 20:54:02,618 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfe0529baac322a72dbf6ef88dd5ff1c/C of bfe0529baac322a72dbf6ef88dd5ff1c into d1e1a4cea29247ca9bb53043ff867f83(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:02,618 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:02,618 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c., storeName=bfe0529baac322a72dbf6ef88dd5ff1c/C, priority=13, startTime=1681678441769; duration=0sec 2023-04-16 20:54:02,618 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:02,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:54:02,676 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-16 20:54:02,676 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:54578 2023-04-16 20:54:02,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:54:02,676 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:02,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:54:02,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:54:02,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:54:02,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:54:02,677 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:54:02,679 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:54578 2023-04-16 20:54:02,679 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:02,680 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77648431 to 127.0.0.1:54578 2023-04-16 20:54:02,680 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:02,683 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b886c8f to 127.0.0.1:54578 2023-04-16 20:54:02,683 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:03,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=873 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/52106c79eb6143989f5b2122c2439db2 2023-04-16 20:54:03,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=873 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/57771c8d0d23426889bbcbc4bd8eaa76 2023-04-16 20:54:03,907 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=873 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/a5cb32ed42234419942905ca0485b74d 2023-04-16 20:54:03,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/52106c79eb6143989f5b2122c2439db2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/52106c79eb6143989f5b2122c2439db2 2023-04-16 20:54:03,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/52106c79eb6143989f5b2122c2439db2, entries=150, sequenceid=873, filesize=11.9 K 2023-04-16 20:54:03,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/57771c8d0d23426889bbcbc4bd8eaa76 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/57771c8d0d23426889bbcbc4bd8eaa76 2023-04-16 20:54:03,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/57771c8d0d23426889bbcbc4bd8eaa76, entries=150, sequenceid=873, filesize=11.9 K 2023-04-16 20:54:03,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/a5cb32ed42234419942905ca0485b74d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a5cb32ed42234419942905ca0485b74d 2023-04-16 20:54:03,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a5cb32ed42234419942905ca0485b74d, entries=150, sequenceid=873, filesize=11.9 K 2023-04-16 20:54:03,925 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=20.13 KB/20610 for bfe0529baac322a72dbf6ef88dd5ff1c in 1249ms, sequenceid=873, compaction requested=false 2023-04-16 20:54:03,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:05,339 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09a575b8 to 127.0.0.1:54578 2023-04-16 20:54:05,340 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 174 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 149 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 151 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 156 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 4013 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 3977 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1770 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5306 rows 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1762 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5282 rows 2023-04-16 20:54:05,340 INFO [Listener at localhost/45179] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 20:54:05,341 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2678d495 to 127.0.0.1:54578 2023-04-16 20:54:05,341 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:05,346 INFO [Listener at localhost/45179] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 20:54:05,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 20:54:05,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:05,359 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678445359"}]},"ts":"1681678445359"} 2023-04-16 20:54:05,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 20:54:05,361 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 20:54:05,364 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 20:54:05,365 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfe0529baac322a72dbf6ef88dd5ff1c, UNASSIGN}] 2023-04-16 20:54:05,367 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfe0529baac322a72dbf6ef88dd5ff1c, UNASSIGN 2023-04-16 20:54:05,367 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=bfe0529baac322a72dbf6ef88dd5ff1c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:05,367 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678445367"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678445367"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678445367"}]},"ts":"1681678445367"} 2023-04-16 20:54:05,369 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:54:05,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 20:54:05,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:54:05,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bfe0529baac322a72dbf6ef88dd5ff1c, disabling compactions & flushes 2023-04-16 20:54:05,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:05,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:05,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. after waiting 0 ms 2023-04-16 20:54:05,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:05,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing bfe0529baac322a72dbf6ef88dd5ff1c 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 20:54:05,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=A 2023-04-16 20:54:05,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:05,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=B 2023-04-16 20:54:05,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:05,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfe0529baac322a72dbf6ef88dd5ff1c, store=C 2023-04-16 20:54:05,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:05,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 20:54:05,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/6df3365ca26a463db001ed1fcde97f92 2023-04-16 20:54:05,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 20:54:06,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/97792ef3212e4c288b5e4fcba11f0571 2023-04-16 20:54:06,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 20:54:06,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/3cac5578932c43eab5f331d0accb14f1 2023-04-16 20:54:06,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/A/6df3365ca26a463db001ed1fcde97f92 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6df3365ca26a463db001ed1fcde97f92 2023-04-16 20:54:06,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6df3365ca26a463db001ed1fcde97f92, entries=100, sequenceid=880, filesize=9.5 K 2023-04-16 20:54:06,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/B/97792ef3212e4c288b5e4fcba11f0571 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/97792ef3212e4c288b5e4fcba11f0571 2023-04-16 20:54:06,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/97792ef3212e4c288b5e4fcba11f0571, entries=100, sequenceid=880, filesize=9.5 K 2023-04-16 20:54:06,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/.tmp/C/3cac5578932c43eab5f331d0accb14f1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3cac5578932c43eab5f331d0accb14f1 2023-04-16 20:54:06,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3cac5578932c43eab5f331d0accb14f1, entries=100, sequenceid=880, filesize=9.5 K 2023-04-16 20:54:06,774 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for bfe0529baac322a72dbf6ef88dd5ff1c in 1247ms, sequenceid=880, compaction requested=true 2023-04-16 20:54:06,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/771aea0c8f1f435f9add8dcb37b02c11, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6a7fa4c7900b488eb5dcc415309fbb39, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1892409c287148feaad8141461d04a38, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5814f3b24e5b44d49e7430751ee3deda, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5547f4488857420a972492f8569e8b33, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c10bcd44c0b7456aa1617f5ac675ddb3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c137c5010e1b46548e3306d5017d3fc7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/bc4a5df7870443d3b2b3bfa74ff7773d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/137d9bf6306d4179a7991bb6ce3acbfb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/be19d543fc5f4077b89c906df5b71870, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ecf049a0540645faa43f4614bb24b802, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/57a54635217d41a4962b07af89c3acf6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f4ea32898f504c24b4d1896fc4e97104, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6660950e49394dac87fc6769f3eafbf0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/211ef114d2b54bcfa977adfff9074761, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f376b60ebedf491e97c886ba8088e158, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/e5b4e9cbb58040e8a7274fb7cfcf59c3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/3a8c50bd92984fa3b57c4a4625890690, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fb39135d43c74befb41bf8ddf67ab2d6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/22bec2b28f80444e841dd6db7ee51ea0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/db57cd3498b3406dbed04fc2a466fd1e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/05144ed31f864e2ab3e1eaac30531976, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8c3854c29d5c40b7a68ea6daa4f562de, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/812f1ef2522048b1b021b04bf9a59bfb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/dc0957be84fe42a8acf17d406846a8ca, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/083f85fa9cce4956b29d4226c88673f1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9b67ee67aaaa4198bcee1f3200f71774, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7dfe88fce0b44b22a873b48c62f82ae4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f5d36474a7374f56ab84b7f5a3bd5bd1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/865bd4dc82704a85a6956ed39a11d782, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7e37a0579ba1480f95d2b7cb9c32301e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fcb8bcacb4cd44b3a77d349b26e753d2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1cca72c30c5941c0a04ff4982168ea5d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/16113580032c4ede9ac7b76bff6338c7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/da35140ec13d49cca58d4d8d9264e814, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ff4d0a124a23434aaeb3cbd28fc55ab9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c2317acf57434a05acb3e72ec6e1c53a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/393fe2d0674d4c1089815a1aa72a5ab8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f8e3cf7c04b542ecbc0b6d133d3e2a92, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9ed8b43249be4e42893b8c0b265fb465, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8099355338ce4dd193bda154583cd895, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5b1bf1200c754ebbb84604fabb615b62] to archive 2023-04-16 20:54:06,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:54:06,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/771aea0c8f1f435f9add8dcb37b02c11 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/771aea0c8f1f435f9add8dcb37b02c11 2023-04-16 20:54:06,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6a7fa4c7900b488eb5dcc415309fbb39 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6a7fa4c7900b488eb5dcc415309fbb39 2023-04-16 20:54:06,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1892409c287148feaad8141461d04a38 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1892409c287148feaad8141461d04a38 2023-04-16 20:54:06,803 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5814f3b24e5b44d49e7430751ee3deda to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5814f3b24e5b44d49e7430751ee3deda 2023-04-16 20:54:06,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5547f4488857420a972492f8569e8b33 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5547f4488857420a972492f8569e8b33 2023-04-16 20:54:06,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c10bcd44c0b7456aa1617f5ac675ddb3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c10bcd44c0b7456aa1617f5ac675ddb3 2023-04-16 20:54:06,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c137c5010e1b46548e3306d5017d3fc7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c137c5010e1b46548e3306d5017d3fc7 2023-04-16 20:54:06,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/bc4a5df7870443d3b2b3bfa74ff7773d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/bc4a5df7870443d3b2b3bfa74ff7773d 2023-04-16 20:54:06,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/137d9bf6306d4179a7991bb6ce3acbfb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/137d9bf6306d4179a7991bb6ce3acbfb 2023-04-16 20:54:06,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/be19d543fc5f4077b89c906df5b71870 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/be19d543fc5f4077b89c906df5b71870 2023-04-16 20:54:06,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ecf049a0540645faa43f4614bb24b802 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ecf049a0540645faa43f4614bb24b802 2023-04-16 20:54:06,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/57a54635217d41a4962b07af89c3acf6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/57a54635217d41a4962b07af89c3acf6 2023-04-16 20:54:06,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f4ea32898f504c24b4d1896fc4e97104 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f4ea32898f504c24b4d1896fc4e97104 2023-04-16 20:54:06,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6660950e49394dac87fc6769f3eafbf0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6660950e49394dac87fc6769f3eafbf0 2023-04-16 20:54:06,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/211ef114d2b54bcfa977adfff9074761 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/211ef114d2b54bcfa977adfff9074761 2023-04-16 20:54:06,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f376b60ebedf491e97c886ba8088e158 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f376b60ebedf491e97c886ba8088e158 2023-04-16 20:54:06,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/e5b4e9cbb58040e8a7274fb7cfcf59c3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/e5b4e9cbb58040e8a7274fb7cfcf59c3 2023-04-16 20:54:06,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/3a8c50bd92984fa3b57c4a4625890690 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/3a8c50bd92984fa3b57c4a4625890690 2023-04-16 20:54:06,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fb39135d43c74befb41bf8ddf67ab2d6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fb39135d43c74befb41bf8ddf67ab2d6 2023-04-16 20:54:06,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/22bec2b28f80444e841dd6db7ee51ea0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/22bec2b28f80444e841dd6db7ee51ea0 2023-04-16 20:54:06,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/db57cd3498b3406dbed04fc2a466fd1e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/db57cd3498b3406dbed04fc2a466fd1e 2023-04-16 20:54:06,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/05144ed31f864e2ab3e1eaac30531976 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/05144ed31f864e2ab3e1eaac30531976 2023-04-16 20:54:06,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8c3854c29d5c40b7a68ea6daa4f562de to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8c3854c29d5c40b7a68ea6daa4f562de 2023-04-16 20:54:06,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/812f1ef2522048b1b021b04bf9a59bfb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/812f1ef2522048b1b021b04bf9a59bfb 2023-04-16 20:54:06,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/dc0957be84fe42a8acf17d406846a8ca to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/dc0957be84fe42a8acf17d406846a8ca 2023-04-16 20:54:06,828 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/083f85fa9cce4956b29d4226c88673f1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/083f85fa9cce4956b29d4226c88673f1 2023-04-16 20:54:06,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9b67ee67aaaa4198bcee1f3200f71774 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9b67ee67aaaa4198bcee1f3200f71774 2023-04-16 20:54:06,831 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7dfe88fce0b44b22a873b48c62f82ae4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7dfe88fce0b44b22a873b48c62f82ae4 2023-04-16 20:54:06,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f5d36474a7374f56ab84b7f5a3bd5bd1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f5d36474a7374f56ab84b7f5a3bd5bd1 2023-04-16 20:54:06,834 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/865bd4dc82704a85a6956ed39a11d782 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/865bd4dc82704a85a6956ed39a11d782 2023-04-16 20:54:06,836 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7e37a0579ba1480f95d2b7cb9c32301e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/7e37a0579ba1480f95d2b7cb9c32301e 2023-04-16 20:54:06,837 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fcb8bcacb4cd44b3a77d349b26e753d2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/fcb8bcacb4cd44b3a77d349b26e753d2 2023-04-16 20:54:06,838 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1cca72c30c5941c0a04ff4982168ea5d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/1cca72c30c5941c0a04ff4982168ea5d 2023-04-16 20:54:06,840 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/16113580032c4ede9ac7b76bff6338c7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/16113580032c4ede9ac7b76bff6338c7 2023-04-16 20:54:06,841 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/da35140ec13d49cca58d4d8d9264e814 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/da35140ec13d49cca58d4d8d9264e814 2023-04-16 20:54:06,842 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ff4d0a124a23434aaeb3cbd28fc55ab9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/ff4d0a124a23434aaeb3cbd28fc55ab9 2023-04-16 20:54:06,843 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c2317acf57434a05acb3e72ec6e1c53a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/c2317acf57434a05acb3e72ec6e1c53a 2023-04-16 20:54:06,844 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/393fe2d0674d4c1089815a1aa72a5ab8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/393fe2d0674d4c1089815a1aa72a5ab8 2023-04-16 20:54:06,845 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f8e3cf7c04b542ecbc0b6d133d3e2a92 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/f8e3cf7c04b542ecbc0b6d133d3e2a92 2023-04-16 20:54:06,847 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9ed8b43249be4e42893b8c0b265fb465 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/9ed8b43249be4e42893b8c0b265fb465 2023-04-16 20:54:06,848 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8099355338ce4dd193bda154583cd895 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/8099355338ce4dd193bda154583cd895 2023-04-16 20:54:06,849 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5b1bf1200c754ebbb84604fabb615b62 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/5b1bf1200c754ebbb84604fabb615b62 2023-04-16 20:54:06,880 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/653206bc1a1a48c29dfb02a6f3fee84d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1baa6fa1b65b41f7a25211563987dd3d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/4cb5ad0846924860bb505e90d509320b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9b09ed514ecc49cdbf3c02c3180bdea2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6ee1c9feb3d4979997c1d3659bf6280, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/34d8daa1e8124b289628dc4c699931f8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/feed592d0d064a3fa2afa3914bced109, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/2712c722476e4050ae6703f1c5a1dbf0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/be62e4ebb62340dd8c367c8e3162791f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1962257b0d3a48e4b6990f9a06a54a6a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d43d9c5d66634f20b4c5918f47d8c84c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/eb866f34ae2848ba9d89b653b4906dff, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/af0a4d273ac9452db751ff6387132e05, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/810fa7544b724eac8d90abd6ed3ceb2a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6dfd54efd3d49dbb5ece086d8cf6d2d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/468b07659f5143f292d313b2aea6c688, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aa5621fac86644eca776883904001a95, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/b70d4593eceb474c949169ff207f4c57, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a8d8f6fca5e6471ebc5e5083b75750f9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d2f3355d5e754b9a97d18cec037e0091, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1afc3e0b355d4cbfae26175cfadf2ee2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7a37f10458e94127ae541d6dd24aeb95, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fc10ea6f413946e4bf15533fffde5e0d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a6f81b2ad7cb47f280bf8138478b1731, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fd91825e894b4c2d8f08d6284399f003, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/6dc435c11e514836864862b69a699e20, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/48c92263b9cf4f829c7d22213869bc2d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9366a0f26ece47f58ec9252d2d8ca781, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d341604a80d34a8b934a2e536c0f3d3b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aeb1e1b564164d109de9d9ecfb2a6194, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9a20dc96ab334ce0b87c61be72cd6bcd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/80f53b494fdc47a99847c41261dac8f5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/69642633f7434f9c9f9f3dd159a56037, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/857e7b764fe84c55974560bc9b8cfb22, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9bd4e55e17a64e8d8e9c7666b73492e5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7886bb9feb53469ab7ffecfb4c81e132, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/21724f73a39f40a3a4d0ab4fa63b10bb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1a4724f5e49e4eb2bd4004ed55eb4c9f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d340b7416a6c4247b1d2d55e1dfadb3b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3cb81e5695784110a1847549dd84cb30, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3dd16b462bd6483897f15ece881f19b6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/0ec53684f9004d3abc8f95e63a57d37d] to archive 2023-04-16 20:54:06,881 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:54:06,883 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/653206bc1a1a48c29dfb02a6f3fee84d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/653206bc1a1a48c29dfb02a6f3fee84d 2023-04-16 20:54:06,885 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1baa6fa1b65b41f7a25211563987dd3d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1baa6fa1b65b41f7a25211563987dd3d 2023-04-16 20:54:06,886 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/4cb5ad0846924860bb505e90d509320b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/4cb5ad0846924860bb505e90d509320b 2023-04-16 20:54:06,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9b09ed514ecc49cdbf3c02c3180bdea2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9b09ed514ecc49cdbf3c02c3180bdea2 2023-04-16 20:54:06,888 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6ee1c9feb3d4979997c1d3659bf6280 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6ee1c9feb3d4979997c1d3659bf6280 2023-04-16 20:54:06,889 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/34d8daa1e8124b289628dc4c699931f8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/34d8daa1e8124b289628dc4c699931f8 2023-04-16 20:54:06,890 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/feed592d0d064a3fa2afa3914bced109 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/feed592d0d064a3fa2afa3914bced109 2023-04-16 20:54:06,891 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/2712c722476e4050ae6703f1c5a1dbf0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/2712c722476e4050ae6703f1c5a1dbf0 2023-04-16 20:54:06,892 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/be62e4ebb62340dd8c367c8e3162791f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/be62e4ebb62340dd8c367c8e3162791f 2023-04-16 20:54:06,894 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1962257b0d3a48e4b6990f9a06a54a6a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1962257b0d3a48e4b6990f9a06a54a6a 2023-04-16 20:54:06,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d43d9c5d66634f20b4c5918f47d8c84c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d43d9c5d66634f20b4c5918f47d8c84c 2023-04-16 20:54:06,897 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/eb866f34ae2848ba9d89b653b4906dff to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/eb866f34ae2848ba9d89b653b4906dff 2023-04-16 20:54:06,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/af0a4d273ac9452db751ff6387132e05 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/af0a4d273ac9452db751ff6387132e05 2023-04-16 20:54:06,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/810fa7544b724eac8d90abd6ed3ceb2a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/810fa7544b724eac8d90abd6ed3ceb2a 2023-04-16 20:54:06,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6dfd54efd3d49dbb5ece086d8cf6d2d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/f6dfd54efd3d49dbb5ece086d8cf6d2d 2023-04-16 20:54:06,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/468b07659f5143f292d313b2aea6c688 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/468b07659f5143f292d313b2aea6c688 2023-04-16 20:54:06,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aa5621fac86644eca776883904001a95 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aa5621fac86644eca776883904001a95 2023-04-16 20:54:06,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/b70d4593eceb474c949169ff207f4c57 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/b70d4593eceb474c949169ff207f4c57 2023-04-16 20:54:06,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a8d8f6fca5e6471ebc5e5083b75750f9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a8d8f6fca5e6471ebc5e5083b75750f9 2023-04-16 20:54:06,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d2f3355d5e754b9a97d18cec037e0091 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d2f3355d5e754b9a97d18cec037e0091 2023-04-16 20:54:06,907 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1afc3e0b355d4cbfae26175cfadf2ee2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1afc3e0b355d4cbfae26175cfadf2ee2 2023-04-16 20:54:06,908 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7a37f10458e94127ae541d6dd24aeb95 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7a37f10458e94127ae541d6dd24aeb95 2023-04-16 20:54:06,909 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fc10ea6f413946e4bf15533fffde5e0d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fc10ea6f413946e4bf15533fffde5e0d 2023-04-16 20:54:06,910 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a6f81b2ad7cb47f280bf8138478b1731 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/a6f81b2ad7cb47f280bf8138478b1731 2023-04-16 20:54:06,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fd91825e894b4c2d8f08d6284399f003 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/fd91825e894b4c2d8f08d6284399f003 2023-04-16 20:54:06,912 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/6dc435c11e514836864862b69a699e20 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/6dc435c11e514836864862b69a699e20 2023-04-16 20:54:06,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/48c92263b9cf4f829c7d22213869bc2d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/48c92263b9cf4f829c7d22213869bc2d 2023-04-16 20:54:06,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9366a0f26ece47f58ec9252d2d8ca781 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9366a0f26ece47f58ec9252d2d8ca781 2023-04-16 20:54:06,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d341604a80d34a8b934a2e536c0f3d3b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d341604a80d34a8b934a2e536c0f3d3b 2023-04-16 20:54:06,916 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aeb1e1b564164d109de9d9ecfb2a6194 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/aeb1e1b564164d109de9d9ecfb2a6194 2023-04-16 20:54:06,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9a20dc96ab334ce0b87c61be72cd6bcd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9a20dc96ab334ce0b87c61be72cd6bcd 2023-04-16 20:54:06,918 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/80f53b494fdc47a99847c41261dac8f5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/80f53b494fdc47a99847c41261dac8f5 2023-04-16 20:54:06,920 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/69642633f7434f9c9f9f3dd159a56037 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/69642633f7434f9c9f9f3dd159a56037 2023-04-16 20:54:06,921 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/857e7b764fe84c55974560bc9b8cfb22 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/857e7b764fe84c55974560bc9b8cfb22 2023-04-16 20:54:06,921 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9bd4e55e17a64e8d8e9c7666b73492e5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/9bd4e55e17a64e8d8e9c7666b73492e5 2023-04-16 20:54:06,922 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7886bb9feb53469ab7ffecfb4c81e132 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7886bb9feb53469ab7ffecfb4c81e132 2023-04-16 20:54:06,923 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/21724f73a39f40a3a4d0ab4fa63b10bb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/21724f73a39f40a3a4d0ab4fa63b10bb 2023-04-16 20:54:06,924 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1a4724f5e49e4eb2bd4004ed55eb4c9f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/1a4724f5e49e4eb2bd4004ed55eb4c9f 2023-04-16 20:54:06,925 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d340b7416a6c4247b1d2d55e1dfadb3b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/d340b7416a6c4247b1d2d55e1dfadb3b 2023-04-16 20:54:06,926 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3cb81e5695784110a1847549dd84cb30 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3cb81e5695784110a1847549dd84cb30 2023-04-16 20:54:06,927 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3dd16b462bd6483897f15ece881f19b6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/3dd16b462bd6483897f15ece881f19b6 2023-04-16 20:54:06,927 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/0ec53684f9004d3abc8f95e63a57d37d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/0ec53684f9004d3abc8f95e63a57d37d 2023-04-16 20:54:06,931 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/82a5f07124c54708973e89534e736a15, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d0e7d5bc6dd64a139cca0dcda20c669e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/68dab8a75b3b40bbba9710adfbdb948b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f385e17007844618c3fea5c3983908f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3f880bd189fa4949af26caac844296e4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6e3ca267fe05482c84c67369e4d738a0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f4fef4a71814b4ba08cf9c9ee16c620, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/825fbe89ca634aedae01c63c9e456d36, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/e497b7d46e2a4e61bc24c1ee5e33206b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/fccc3c12122c47aba70abf3716e7954f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/8b64791bfecd4e8fb74e02af65022556, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2236a062809041f0929630733cc051fd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/24a6f3cd439e40f786ee255e5d02c0e5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dc6c95c8dabc4918b6f15d201bf5d9fc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/65f6889460a640f284149f8d45991b12, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7cebae6b18dd44bc91f5c93959b4ba78, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/40e31a6608e3474daeeb3ca9ddb12e2c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a1ae1df1c81a4bd79f461a392f70e211, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/0551857176174541961a8162c098cfd0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/93ca639ddd204b098d0ebf2f94f88158, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3033d918c6d84c3ab7e7ab635206f628, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ffe8ae1ec8df462d89cd5c2cb39934bb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dbb9b82239cf4bd79c6ae8a1a8a8a05a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/bc6eaacd80664343a3c76ebbfa23ceb5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4fe8177a2145441d95f867f3b452981f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3a0deae3620a41bdaacfc74f729e0855, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/154095ff0f874b9098f40f3f63a13cea, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4ff35fe71ad74d079b0160df233adeae, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d44dc21298594b4b97ffca875f90e7ee, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ca9c8c1f3f3042a18904bd14ac3b29c7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/18f909a3b5ef4e5b88d6ee81761e6d77, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/08a9048508cd43ae8c4e38ce9a4e3c3c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d343ada7218a4a518cc91d47ee41ea0c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/b1dcf555d605427d8f23fd314a17f5d9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d6baa6dbf0af4383ba2e6111213cf9a4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/01a1bf4071f645cf859cc74944c73ac6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2086b9c5aff647dd96b675b9acd9efb8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a7b6912feb6f41488f479d0ffe1ad5be, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f9ae11b0df5b4d57a587cbccde3c6384, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7006bae1e56546da929ecdcff12b547d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/aadab7472118421ea617ce4fe62d5422, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f35de83c862b47699070e8deac8af058] to archive 2023-04-16 20:54:06,932 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:54:06,933 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/82a5f07124c54708973e89534e736a15 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/82a5f07124c54708973e89534e736a15 2023-04-16 20:54:06,934 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d0e7d5bc6dd64a139cca0dcda20c669e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d0e7d5bc6dd64a139cca0dcda20c669e 2023-04-16 20:54:06,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/68dab8a75b3b40bbba9710adfbdb948b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/68dab8a75b3b40bbba9710adfbdb948b 2023-04-16 20:54:06,936 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f385e17007844618c3fea5c3983908f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f385e17007844618c3fea5c3983908f 2023-04-16 20:54:06,937 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3f880bd189fa4949af26caac844296e4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3f880bd189fa4949af26caac844296e4 2023-04-16 20:54:06,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6e3ca267fe05482c84c67369e4d738a0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6e3ca267fe05482c84c67369e4d738a0 2023-04-16 20:54:06,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f4fef4a71814b4ba08cf9c9ee16c620 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/6f4fef4a71814b4ba08cf9c9ee16c620 2023-04-16 20:54:06,940 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/825fbe89ca634aedae01c63c9e456d36 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/825fbe89ca634aedae01c63c9e456d36 2023-04-16 20:54:06,941 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/e497b7d46e2a4e61bc24c1ee5e33206b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/e497b7d46e2a4e61bc24c1ee5e33206b 2023-04-16 20:54:06,942 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/fccc3c12122c47aba70abf3716e7954f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/fccc3c12122c47aba70abf3716e7954f 2023-04-16 20:54:06,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/8b64791bfecd4e8fb74e02af65022556 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/8b64791bfecd4e8fb74e02af65022556 2023-04-16 20:54:06,944 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2236a062809041f0929630733cc051fd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2236a062809041f0929630733cc051fd 2023-04-16 20:54:06,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/24a6f3cd439e40f786ee255e5d02c0e5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/24a6f3cd439e40f786ee255e5d02c0e5 2023-04-16 20:54:06,946 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dc6c95c8dabc4918b6f15d201bf5d9fc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dc6c95c8dabc4918b6f15d201bf5d9fc 2023-04-16 20:54:06,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/65f6889460a640f284149f8d45991b12 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/65f6889460a640f284149f8d45991b12 2023-04-16 20:54:06,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7cebae6b18dd44bc91f5c93959b4ba78 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7cebae6b18dd44bc91f5c93959b4ba78 2023-04-16 20:54:06,949 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/40e31a6608e3474daeeb3ca9ddb12e2c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/40e31a6608e3474daeeb3ca9ddb12e2c 2023-04-16 20:54:06,950 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a1ae1df1c81a4bd79f461a392f70e211 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a1ae1df1c81a4bd79f461a392f70e211 2023-04-16 20:54:06,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/0551857176174541961a8162c098cfd0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/0551857176174541961a8162c098cfd0 2023-04-16 20:54:06,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/93ca639ddd204b098d0ebf2f94f88158 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/93ca639ddd204b098d0ebf2f94f88158 2023-04-16 20:54:06,952 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3033d918c6d84c3ab7e7ab635206f628 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3033d918c6d84c3ab7e7ab635206f628 2023-04-16 20:54:06,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ffe8ae1ec8df462d89cd5c2cb39934bb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ffe8ae1ec8df462d89cd5c2cb39934bb 2023-04-16 20:54:06,954 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dbb9b82239cf4bd79c6ae8a1a8a8a05a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/dbb9b82239cf4bd79c6ae8a1a8a8a05a 2023-04-16 20:54:06,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/bc6eaacd80664343a3c76ebbfa23ceb5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/bc6eaacd80664343a3c76ebbfa23ceb5 2023-04-16 20:54:06,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4fe8177a2145441d95f867f3b452981f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4fe8177a2145441d95f867f3b452981f 2023-04-16 20:54:06,957 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3a0deae3620a41bdaacfc74f729e0855 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3a0deae3620a41bdaacfc74f729e0855 2023-04-16 20:54:06,958 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/154095ff0f874b9098f40f3f63a13cea to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/154095ff0f874b9098f40f3f63a13cea 2023-04-16 20:54:06,959 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4ff35fe71ad74d079b0160df233adeae to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/4ff35fe71ad74d079b0160df233adeae 2023-04-16 20:54:06,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d44dc21298594b4b97ffca875f90e7ee to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d44dc21298594b4b97ffca875f90e7ee 2023-04-16 20:54:06,962 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ca9c8c1f3f3042a18904bd14ac3b29c7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/ca9c8c1f3f3042a18904bd14ac3b29c7 2023-04-16 20:54:06,963 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/18f909a3b5ef4e5b88d6ee81761e6d77 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/18f909a3b5ef4e5b88d6ee81761e6d77 2023-04-16 20:54:06,964 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/08a9048508cd43ae8c4e38ce9a4e3c3c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/08a9048508cd43ae8c4e38ce9a4e3c3c 2023-04-16 20:54:06,965 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d343ada7218a4a518cc91d47ee41ea0c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d343ada7218a4a518cc91d47ee41ea0c 2023-04-16 20:54:06,966 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/b1dcf555d605427d8f23fd314a17f5d9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/b1dcf555d605427d8f23fd314a17f5d9 2023-04-16 20:54:06,968 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d6baa6dbf0af4383ba2e6111213cf9a4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d6baa6dbf0af4383ba2e6111213cf9a4 2023-04-16 20:54:06,969 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/01a1bf4071f645cf859cc74944c73ac6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/01a1bf4071f645cf859cc74944c73ac6 2023-04-16 20:54:06,970 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2086b9c5aff647dd96b675b9acd9efb8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/2086b9c5aff647dd96b675b9acd9efb8 2023-04-16 20:54:06,971 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a7b6912feb6f41488f479d0ffe1ad5be to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a7b6912feb6f41488f479d0ffe1ad5be 2023-04-16 20:54:06,972 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f9ae11b0df5b4d57a587cbccde3c6384 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f9ae11b0df5b4d57a587cbccde3c6384 2023-04-16 20:54:06,974 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7006bae1e56546da929ecdcff12b547d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/7006bae1e56546da929ecdcff12b547d 2023-04-16 20:54:06,975 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/aadab7472118421ea617ce4fe62d5422 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/aadab7472118421ea617ce4fe62d5422 2023-04-16 20:54:06,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f35de83c862b47699070e8deac8af058 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/f35de83c862b47699070e8deac8af058 2023-04-16 20:54:06,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/recovered.edits/883.seqid, newMaxSeqId=883, maxSeqId=1 2023-04-16 20:54:06,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c. 2023-04-16 20:54:06,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bfe0529baac322a72dbf6ef88dd5ff1c: 2023-04-16 20:54:06,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:54:06,984 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=bfe0529baac322a72dbf6ef88dd5ff1c, regionState=CLOSED 2023-04-16 20:54:06,985 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678446984"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678446984"}]},"ts":"1681678446984"} 2023-04-16 20:54:06,988 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-16 20:54:06,988 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure bfe0529baac322a72dbf6ef88dd5ff1c, server=jenkins-hbase4.apache.org,41211,1681678416093 in 1.6170 sec 2023-04-16 20:54:06,990 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-16 20:54:06,990 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfe0529baac322a72dbf6ef88dd5ff1c, UNASSIGN in 1.6230 sec 2023-04-16 20:54:06,991 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678446991"}]},"ts":"1681678446991"} 2023-04-16 20:54:06,992 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 20:54:06,994 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 20:54:06,996 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6430 sec 2023-04-16 20:54:07,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 20:54:07,464 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-16 20:54:07,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 20:54:07,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:07,476 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:07,478 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:07,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-16 20:54:07,482 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:54:07,485 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/recovered.edits] 2023-04-16 20:54:07,488 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/3dddaf41997f46ce96dc50a22df65819 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/3dddaf41997f46ce96dc50a22df65819 2023-04-16 20:54:07,490 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/52106c79eb6143989f5b2122c2439db2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/52106c79eb6143989f5b2122c2439db2 2023-04-16 20:54:07,491 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6df3365ca26a463db001ed1fcde97f92 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/A/6df3365ca26a463db001ed1fcde97f92 2023-04-16 20:54:07,494 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/57771c8d0d23426889bbcbc4bd8eaa76 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/57771c8d0d23426889bbcbc4bd8eaa76 2023-04-16 20:54:07,495 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7d3073f046284495ae61878068072fad to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/7d3073f046284495ae61878068072fad 2023-04-16 20:54:07,496 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/97792ef3212e4c288b5e4fcba11f0571 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/B/97792ef3212e4c288b5e4fcba11f0571 2023-04-16 20:54:07,499 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3cac5578932c43eab5f331d0accb14f1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/3cac5578932c43eab5f331d0accb14f1 2023-04-16 20:54:07,500 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a5cb32ed42234419942905ca0485b74d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/a5cb32ed42234419942905ca0485b74d 2023-04-16 20:54:07,502 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d1e1a4cea29247ca9bb53043ff867f83 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/C/d1e1a4cea29247ca9bb53043ff867f83 2023-04-16 20:54:07,505 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/recovered.edits/883.seqid to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c/recovered.edits/883.seqid 2023-04-16 20:54:07,506 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/bfe0529baac322a72dbf6ef88dd5ff1c 2023-04-16 20:54:07,506 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:54:07,509 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:07,531 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 20:54:07,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-16 20:54:07,583 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 20:54:07,585 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:07,585 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 20:54:07,585 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681678447585"}]},"ts":"9223372036854775807"} 2023-04-16 20:54:07,587 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 20:54:07,587 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bfe0529baac322a72dbf6ef88dd5ff1c, NAME => 'TestAcidGuarantees,,1681678419722.bfe0529baac322a72dbf6ef88dd5ff1c.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 20:54:07,587 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 20:54:07,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681678447587"}]},"ts":"9223372036854775807"} 2023-04-16 20:54:07,590 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 20:54:07,592 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:07,593 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 122 msec 2023-04-16 20:54:07,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-16 20:54:07,781 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-16 20:54:07,797 INFO [Listener at localhost/45179] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1e748436-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-163506051_15 at /127.0.0.1:58832 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1e748436-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1e748436-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1e748436-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:41211-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-163506051_15 at /127.0.0.1:58906 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=588 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=408 (was 285) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=706 (was 1407) 2023-04-16 20:54:07,807 INFO [Listener at localhost/45179] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=588, MaxFileDescriptor=60000, SystemLoadAverage=408, ProcessCount=169, AvailableMemoryMB=706 2023-04-16 20:54:07,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 20:54:07,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:54:07,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:07,812 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 20:54:07,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-16 20:54:07,813 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 20:54:07,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 20:54:07,814 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:07,815 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16 empty. 2023-04-16 20:54:07,815 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:07,815 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:54:07,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 20:54:08,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 20:54:08,223 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 20:54:08,224 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => fb9117f6cdea5099d0a91a955c183b16, NAME => 'TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp 2023-04-16 20:54:08,230 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:54:08,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing fb9117f6cdea5099d0a91a955c183b16, disabling compactions & flushes 2023-04-16 20:54:08,231 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:08,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:08,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. after waiting 0 ms 2023-04-16 20:54:08,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:08,231 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:08,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:08,233 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 20:54:08,234 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678448234"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678448234"}]},"ts":"1681678448234"} 2023-04-16 20:54:08,236 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 20:54:08,237 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 20:54:08,237 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678448237"}]},"ts":"1681678448237"} 2023-04-16 20:54:08,238 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 20:54:08,241 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, ASSIGN}] 2023-04-16 20:54:08,243 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, ASSIGN 2023-04-16 20:54:08,243 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=false 2023-04-16 20:54:08,394 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=fb9117f6cdea5099d0a91a955c183b16, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:08,395 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678448394"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678448394"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678448394"}]},"ts":"1681678448394"} 2023-04-16 20:54:08,396 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:54:08,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 20:54:08,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:08,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fb9117f6cdea5099d0a91a955c183b16, NAME => 'TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:54:08,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:08,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:54:08,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:08,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:08,553 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:08,555 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A 2023-04-16 20:54:08,555 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A 2023-04-16 20:54:08,555 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:54:08,555 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb9117f6cdea5099d0a91a955c183b16 columnFamilyName A 2023-04-16 20:54:08,556 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(310): Store=fb9117f6cdea5099d0a91a955c183b16/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:54:08,556 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:08,557 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B 2023-04-16 20:54:08,557 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B 2023-04-16 20:54:08,557 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:54:08,557 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb9117f6cdea5099d0a91a955c183b16 columnFamilyName B 2023-04-16 20:54:08,558 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(310): Store=fb9117f6cdea5099d0a91a955c183b16/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:54:08,558 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:08,559 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C 2023-04-16 20:54:08,559 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C 2023-04-16 20:54:08,559 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:54:08,559 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb9117f6cdea5099d0a91a955c183b16 columnFamilyName C 2023-04-16 20:54:08,560 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(310): Store=fb9117f6cdea5099d0a91a955c183b16/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:54:08,560 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:08,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:08,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:08,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:54:08,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:08,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 20:54:08,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fb9117f6cdea5099d0a91a955c183b16; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67731735, jitterRate=0.009281501173973083}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:54:08,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:08,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., pid=18, masterSystemTime=1681678448548 2023-04-16 20:54:08,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:08,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:08,568 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=fb9117f6cdea5099d0a91a955c183b16, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:08,568 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678448568"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678448568"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678448568"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678448568"}]},"ts":"1681678448568"} 2023-04-16 20:54:08,571 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-16 20:54:08,571 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 in 174 msec 2023-04-16 20:54:08,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-16 20:54:08,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, ASSIGN in 330 msec 2023-04-16 20:54:08,573 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 20:54:08,573 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678448573"}]},"ts":"1681678448573"} 2023-04-16 20:54:08,574 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 20:54:08,576 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 20:54:08,578 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 767 msec 2023-04-16 20:54:08,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 20:54:08,917 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-16 20:54:08,918 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x52274650 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:08,927 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7829b43e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:08,928 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:08,933 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59846, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:08,934 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 20:54:08,941 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34622, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 20:54:08,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 20:54:08,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:54:08,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:09,376 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 20:54:09,377 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 20:54:09,377 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 20:54:09,381 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-16 20:54:09,393 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, REOPEN/MOVE}] 2023-04-16 20:54:09,394 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, REOPEN/MOVE 2023-04-16 20:54:09,394 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fb9117f6cdea5099d0a91a955c183b16, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:09,394 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678449394"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678449394"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678449394"}]},"ts":"1681678449394"} 2023-04-16 20:54:09,396 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:54:09,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fb9117f6cdea5099d0a91a955c183b16, disabling compactions & flushes 2023-04-16 20:54:09,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:09,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:09,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. after waiting 0 ms 2023-04-16 20:54:09,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:09,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-16 20:54:09,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:09,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:09,553 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: fb9117f6cdea5099d0a91a955c183b16 to self. 2023-04-16 20:54:09,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,555 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fb9117f6cdea5099d0a91a955c183b16, regionState=CLOSED 2023-04-16 20:54:09,555 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678449555"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678449555"}]},"ts":"1681678449555"} 2023-04-16 20:54:09,558 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-16 20:54:09,558 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 in 160 msec 2023-04-16 20:54:09,558 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=true 2023-04-16 20:54:09,709 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fb9117f6cdea5099d0a91a955c183b16, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:09,709 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678449709"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678449709"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678449709"}]},"ts":"1681678449709"} 2023-04-16 20:54:09,711 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:54:09,868 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:09,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fb9117f6cdea5099d0a91a955c183b16, NAME => 'TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:54:09,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:54:09,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,871 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,872 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A 2023-04-16 20:54:09,872 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A 2023-04-16 20:54:09,873 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:54:09,876 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb9117f6cdea5099d0a91a955c183b16 columnFamilyName A 2023-04-16 20:54:09,878 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(310): Store=fb9117f6cdea5099d0a91a955c183b16/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:54:09,879 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,880 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B 2023-04-16 20:54:09,880 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B 2023-04-16 20:54:09,880 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:54:09,880 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb9117f6cdea5099d0a91a955c183b16 columnFamilyName B 2023-04-16 20:54:09,881 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(310): Store=fb9117f6cdea5099d0a91a955c183b16/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:54:09,881 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,882 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C 2023-04-16 20:54:09,882 DEBUG [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C 2023-04-16 20:54:09,882 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:54:09,882 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb9117f6cdea5099d0a91a955c183b16 columnFamilyName C 2023-04-16 20:54:09,883 INFO [StoreOpener-fb9117f6cdea5099d0a91a955c183b16-1] regionserver.HStore(310): Store=fb9117f6cdea5099d0a91a955c183b16/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:54:09,883 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:09,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:54:09,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:09,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fb9117f6cdea5099d0a91a955c183b16; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59886396, jitterRate=-0.1076231598854065}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:54:09,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:09,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., pid=23, masterSystemTime=1681678449863 2023-04-16 20:54:09,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:09,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:09,891 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fb9117f6cdea5099d0a91a955c183b16, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:09,891 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678449891"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678449891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678449891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678449891"}]},"ts":"1681678449891"} 2023-04-16 20:54:09,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-16 20:54:09,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 in 182 msec 2023-04-16 20:54:09,896 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-16 20:54:09,896 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, REOPEN/MOVE in 501 msec 2023-04-16 20:54:09,899 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-16 20:54:09,899 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 517 msec 2023-04-16 20:54:09,901 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 946 msec 2023-04-16 20:54:09,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-16 20:54:09,911 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x34135972 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:09,916 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33a47c05, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:09,916 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x02bab6b2 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:09,920 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1441c9f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:09,920 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:09,923 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:09,924 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x370947cb to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:09,929 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c1c92c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:09,930 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:09,933 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:09,933 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x498694cb to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:09,938 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c4379c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:09,938 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:09,942 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:09,943 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:09,948 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:09,949 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:09,952 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:09,956 DEBUG [hconnection-0x244757e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:09,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:09,960 DEBUG [hconnection-0x3cb71ea0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:09,961 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59862, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:09,964 DEBUG [hconnection-0x4ce5f918-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:09,972 DEBUG [hconnection-0x5959edbd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:09,975 DEBUG [hconnection-0xe4ab2a1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:09,975 DEBUG [hconnection-0x208044e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:09,975 DEBUG [hconnection-0x152206e1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:09,977 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59864, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:09,978 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59884, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:09,978 DEBUG [hconnection-0x40e26339-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:09,977 DEBUG [hconnection-0x7a05cdf4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:09,977 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:09,979 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59890, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:09,979 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59906, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:09,981 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:09,989 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59908, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:09,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:09,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:09,989 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:09,989 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59922, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:09,989 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:09,999 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:09,999 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:10,000 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,000 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,003 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:10,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:10,004 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,004 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:10,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:10,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:10,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:10,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:10,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:10,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:10,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:10,006 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:10,006 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:10,006 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:10,006 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:10,006 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-16 20:54:10,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:10,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:10,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:10,014 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:10,014 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:10,014 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:10,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:10,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:10,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678450036, max timestamp=1681678450047Num uniques 859; 2023-04-16 20:54:10,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:10,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678450036, max timestamp=1681678450047Num uniques 859; 2023-04-16 20:54:10,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:10,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678450036, max timestamp=1681678450047Num uniques 859; 2023-04-16 20:54:10,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:10,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678450077, max timestamp=1681678450089Num uniques 1018; 2023-04-16 20:54:10,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 20:54:10,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678450077, max timestamp=1681678450090Num uniques 1018; 2023-04-16 20:54:10,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 20:54:10,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21421, totalHeapSize=48912, min timestamp=1681678450071, max timestamp=1681678450089Num uniques 1018; 2023-04-16 20:54:10,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46611, new segment data size=21421, suffix heap size=133840,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=468 2023-04-16 20:54:10,113 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416de3f767e13054778862e39ae8e9c704c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416de3f767e13054778862e39ae8e9c704c 2023-04-16 20:54:10,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678510118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678510123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678510119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678510124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678510128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,133 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=12, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/3e62900126cd48f5a76ffcd761e851d1 2023-04-16 20:54:10,169 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/ca264b0ca7f44ea3856041cb8c1587cc 2023-04-16 20:54:10,226 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/d7c3dcc9a346491cb1b3d4cef1ea6cfb 2023-04-16 20:54:10,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678510228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678510229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678510233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678510237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/3e62900126cd48f5a76ffcd761e851d1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3e62900126cd48f5a76ffcd761e851d1 2023-04-16 20:54:10,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678510247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,265 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3e62900126cd48f5a76ffcd761e851d1, entries=150, sequenceid=12, filesize=25.1 K 2023-04-16 20:54:10,266 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/ca264b0ca7f44ea3856041cb8c1587cc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/ca264b0ca7f44ea3856041cb8c1587cc 2023-04-16 20:54:10,281 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/ca264b0ca7f44ea3856041cb8c1587cc, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 20:54:10,282 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/d7c3dcc9a346491cb1b3d4cef1ea6cfb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d7c3dcc9a346491cb1b3d4cef1ea6cfb 2023-04-16 20:54:10,287 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d7c3dcc9a346491cb1b3d4cef1ea6cfb, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 20:54:10,288 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for fb9117f6cdea5099d0a91a955c183b16 in 282ms, sequenceid=12, compaction requested=false 2023-04-16 20:54:10,288 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-16 20:54:10,289 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:10,289 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:10,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:10,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:10,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,290 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:10,290 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:10,292 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:10,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:10,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:10,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:10,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff77000[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:10,296 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:10,296 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,298 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,298 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:10,298 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:10,298 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:10,298 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,298 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:10,303 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:10,303 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:10,303 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:10,303 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:10,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:10,308 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:10,308 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:10,308 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:10,308 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:10,308 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:10,309 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,309 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,309 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:10,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:10,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,310 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:10,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:10,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:10,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:10,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:10,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:10,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:10,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:10,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:10,318 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:10,318 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:10,418 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:10,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:10,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:10,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:10,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:10,430 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:10,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:10,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:10,431 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:10,432 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,432 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,433 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:10,433 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:10,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,434 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,434 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:10,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:10,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:10,439 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=154.31 KB heapSize=435.23 KB 2023-04-16 20:54:10,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:10,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:54:10,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:10,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:54:10,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:10,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:54:10,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:10,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678510446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:10,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:10,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:10,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:10,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:10,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678510448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678510448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678510449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:10,460 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:10,461 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:10,461 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:10,461 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678510460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:10,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:10,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:10,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:10,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4972d57e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:10,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:10,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,475 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:10,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:10,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:10,475 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:10,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:10,480 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:10,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,480 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:10,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:10,485 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:10,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:10,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:10,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:10,485 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:10,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:10,487 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:10,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:10,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,492 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:10,492 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:10,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:10,492 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:10,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:10,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:10,493 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:10,493 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:10,492 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:10,492 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:10,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:10,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:10,510 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416bc231370cad24d53b0f374c9f90c465a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bc231370cad24d53b0f374c9f90c465a 2023-04-16 20:54:10,520 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=67, memsize=60.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/6e0fa8fe03954f4ca49e521003b8e5df 2023-04-16 20:54:10,533 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/da5191fd279144ecad1f1c22c59f4fd5 2023-04-16 20:54:10,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678510549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678510550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678510550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/75847f249e1848c1be8ebcb6e718e826 2023-04-16 20:54:10,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/6e0fa8fe03954f4ca49e521003b8e5df as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e0fa8fe03954f4ca49e521003b8e5df 2023-04-16 20:54:10,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e0fa8fe03954f4ca49e521003b8e5df, entries=200, sequenceid=67, filesize=31.9 K 2023-04-16 20:54:10,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/da5191fd279144ecad1f1c22c59f4fd5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/da5191fd279144ecad1f1c22c59f4fd5 2023-04-16 20:54:10,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/da5191fd279144ecad1f1c22c59f4fd5, entries=150, sequenceid=67, filesize=11.6 K 2023-04-16 20:54:10,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/75847f249e1848c1be8ebcb6e718e826 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/75847f249e1848c1be8ebcb6e718e826 2023-04-16 20:54:10,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/75847f249e1848c1be8ebcb6e718e826, entries=150, sequenceid=67, filesize=11.6 K 2023-04-16 20:54:10,582 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~435.19 KB/445632, currentSize=33.54 KB/34350 for fb9117f6cdea5099d0a91a955c183b16 in 143ms, sequenceid=67, compaction requested=false 2023-04-16 20:54:10,584 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:10,593 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:10,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:10,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:10,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:10,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:10,612 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:10,612 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:10,613 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:10,613 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:10,613 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,613 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,614 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,614 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:10,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:10,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:10,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:10,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:10,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:10,633 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:10,633 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:10,633 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-16 20:54:10,634 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:10,634 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:10,634 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:10,634 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:10,635 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:10,635 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:10,701 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304167f1e1611d46846b990db22b44ec17840 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167f1e1611d46846b990db22b44ec17840 2023-04-16 20:54:10,709 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=75, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/e9fe7c707b704b5fb652fc4668f3ff4c 2023-04-16 20:54:10,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:10,755 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:10,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681678450759, max timestamp=1681678450769Num uniques 859; 2023-04-16 20:54:10,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 20:54:10,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681678450759, max timestamp=1681678450769Num uniques 859; 2023-04-16 20:54:10,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 20:54:10,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678510777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678510785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678510785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678510786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678510789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678510891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678510892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678510892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678510893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:10,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:10,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678510893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678511093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678511098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678511098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678511098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678511099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,124 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/20bd66316f5f47398652c75f1764c516 2023-04-16 20:54:11,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678511400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678511402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678511404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678511404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678511405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,536 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/38158b5b40564aedbd4813460054f4eb 2023-04-16 20:54:11,540 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/e9fe7c707b704b5fb652fc4668f3ff4c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e9fe7c707b704b5fb652fc4668f3ff4c 2023-04-16 20:54:11,545 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e9fe7c707b704b5fb652fc4668f3ff4c, entries=150, sequenceid=75, filesize=25.1 K 2023-04-16 20:54:11,546 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/20bd66316f5f47398652c75f1764c516 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/20bd66316f5f47398652c75f1764c516 2023-04-16 20:54:11,550 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/20bd66316f5f47398652c75f1764c516, entries=150, sequenceid=75, filesize=11.6 K 2023-04-16 20:54:11,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/38158b5b40564aedbd4813460054f4eb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/38158b5b40564aedbd4813460054f4eb 2023-04-16 20:54:11,558 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/38158b5b40564aedbd4813460054f4eb, entries=150, sequenceid=75, filesize=11.6 K 2023-04-16 20:54:11,558 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=145.36 KB/148850 for fb9117f6cdea5099d0a91a955c183b16 in 925ms, sequenceid=75, compaction requested=true 2023-04-16 20:54:11,559 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:11,559 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:11,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:11,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:11,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:11,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:11,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,560 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:11,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:11,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:11,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45037be9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:11,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:11,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,567 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:11,568 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,568 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:11,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:11,569 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,569 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:11,569 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:11,569 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:11,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:11,575 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:11,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:11,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:11,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:11,575 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:11,578 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,578 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:11,578 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:11,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:11,579 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:11,584 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:11,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:11,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,584 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:11,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:11,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:11,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:11,585 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:11,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:11,585 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:11,585 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:11,685 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:11,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:11,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:11,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:11,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:11,712 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:11,712 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:11,712 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:11,712 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:11,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,714 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:11,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:11,714 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,716 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,716 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:11,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:11,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:11,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:11,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:11,728 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:11,728 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:11,728 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:11,729 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=145.36 KB heapSize=413.78 KB 2023-04-16 20:54:11,729 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:11,729 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:11,729 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:11,729 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:11,729 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:11,729 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:11,750 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304164445faa1d080427cab3db813e5b2474d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164445faa1d080427cab3db813e5b2474d 2023-04-16 20:54:11,761 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/9dabf7e4a74c4f989fe22d6ec00950cb 2023-04-16 20:54:11,782 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/d30e5455f4604e08bad54e614b9dd957 2023-04-16 20:54:11,796 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/3fbda78375b84df69958a30ab8d37b9b 2023-04-16 20:54:11,801 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/9dabf7e4a74c4f989fe22d6ec00950cb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/9dabf7e4a74c4f989fe22d6ec00950cb 2023-04-16 20:54:11,805 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/9dabf7e4a74c4f989fe22d6ec00950cb, entries=150, sequenceid=107, filesize=25.1 K 2023-04-16 20:54:11,806 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/d30e5455f4604e08bad54e614b9dd957 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/d30e5455f4604e08bad54e614b9dd957 2023-04-16 20:54:11,810 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/d30e5455f4604e08bad54e614b9dd957, entries=150, sequenceid=107, filesize=11.6 K 2023-04-16 20:54:11,811 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/3fbda78375b84df69958a30ab8d37b9b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3fbda78375b84df69958a30ab8d37b9b 2023-04-16 20:54:11,818 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3fbda78375b84df69958a30ab8d37b9b, entries=150, sequenceid=107, filesize=11.6 K 2023-04-16 20:54:11,819 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~145.36 KB/148850, heapSize ~413.73 KB/423664, currentSize=0 B/0 for fb9117f6cdea5099d0a91a955c183b16 in 90ms, sequenceid=107, compaction requested=true 2023-04-16 20:54:11,819 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:11,819 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:11,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:11,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:11,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:11,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:11,822 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,822 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:11,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:11,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791fd807[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:11,825 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:11,825 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,827 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:11,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:11,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:11,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:11,829 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:11,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:11,830 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:11,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:11,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:11,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:11,830 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:11,837 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,837 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:11,837 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:11,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:11,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,839 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,844 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:11,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:11,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:11,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:11,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:11,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:11,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:11,845 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:11,846 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:11,846 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:11,946 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:11,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:11,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:11,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:11,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:11,953 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:11,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:11,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:11,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:11,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,955 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:11,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:11,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:11,956 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:54:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:11,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:11,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:11,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:11,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:11,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:11,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:11,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:11,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:11,960 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:11,960 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:11,960 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:11,960 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:11,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:11,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:11,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:11,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7decd054[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:11,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:11,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,967 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:11,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:11,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:11,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:11,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:11,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:11,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:11,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:11,973 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:11,973 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:11,973 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:11,973 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:11,973 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:11,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:11,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,977 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:11,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:11,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:11,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:11,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:11,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:11,980 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,980 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:11,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,980 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,980 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:11,980 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:11,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:11,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:11,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681678451957, max timestamp=1681678451998Num uniques 859; 2023-04-16 20:54:12,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681678451957, max timestamp=1681678451998Num uniques 859; 2023-04-16 20:54:12,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681678451957, max timestamp=1681678451998Num uniques 859; 2023-04-16 20:54:12,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-16 20:54:12,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-16 20:54:12,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-16 20:54:12,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678512008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678512010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678512011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678512011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678512013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,080 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:12,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:12,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:12,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:12,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:12,084 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:12,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:12,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:12,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,087 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:12,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:12,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:12,089 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,089 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:12,089 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:12,089 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:12,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:12,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226a33b2[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:12,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:12,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,093 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:12,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:12,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:12,094 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,094 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:12,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,095 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,095 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,095 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:12,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:12,097 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:12,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:12,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:12,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:12,098 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:12,100 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,100 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,100 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,101 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,102 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:12,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:12,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:12,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,105 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:12,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,105 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:12,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678512114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678512114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678512116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678512121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678512121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,206 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:12,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:12,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:12,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:12,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:12,215 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:12,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:12,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:12,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:12,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:12,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:12,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:12,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:12,226 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:12,226 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:12,227 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,227 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:12,227 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:12,227 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,227 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:12,227 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:12,227 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,227 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:12,227 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:12,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,229 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,229 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,229 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:12,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c68c825[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:12,233 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:12,233 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,240 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:12,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:12,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:12,240 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,241 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:12,241 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:12,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:12,246 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:12,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:12,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:12,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:12,246 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:12,248 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,248 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,248 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,248 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:12,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:12,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:12,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,256 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,256 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:12,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678512317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678512317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678512326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678512332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678512332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,357 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:12,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:12,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:12,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:12,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:12,360 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:12,360 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:12,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:12,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,362 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:12,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:12,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:12,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:12,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:12,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:12,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:12,367 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:12,367 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:12,367 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,368 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:12,368 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:12,368 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:12,369 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,369 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,370 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,370 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:12,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b30f84f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:12,372 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:12,372 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,374 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:12,374 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:12,374 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:12,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:12,375 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041603f1a38b85a94b5792ae7caf3aee7310 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041603f1a38b85a94b5792ae7caf3aee7310 2023-04-16 20:54:12,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:12,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:12,377 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:12,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:12,380 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:12,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:12,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:12,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:12,380 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:12,382 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,382 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,382 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,384 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,389 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,389 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,389 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:12,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:12,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,389 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:12,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,390 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:12,390 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:12,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,402 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/6e9734464824446f825a9c438f308d7b 2023-04-16 20:54:12,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/24ebcb820b014430aa54d892f7c13fa5 2023-04-16 20:54:12,490 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:12,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:12,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:12,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:12,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:12,494 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:12,495 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,495 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:12,495 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:12,495 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,495 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,496 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,497 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,497 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:12,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:12,498 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,498 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:12,499 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:12,499 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:12,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:12,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:12,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:12,501 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:12,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:12,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c5f90d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:12,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:12,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,504 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:12,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:12,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:12,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,506 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:12,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:12,509 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:12,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:12,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:12,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:12,509 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:12,510 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,510 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,511 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,511 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,512 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:12,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:12,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,516 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,516 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:12,516 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:12,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:12,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,523 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 20:54:12,616 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:12,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:12,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:12,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:12,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:12,620 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:12,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:12,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:12,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,622 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:12,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678512620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:12,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:12,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:12,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678512622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:12,624 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:12,624 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:12,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:12,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:12,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,624 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:12,624 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:12,624 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,624 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:12,624 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:12,625 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,625 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,626 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,626 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:12,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c1a104[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:12,627 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:12,627 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:12,629 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,629 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:12,631 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678512631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:12,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:12,634 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:12,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:12,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:12,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:12,635 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:12,636 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,636 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,636 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,637 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678512634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678512636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,642 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,642 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,642 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,642 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:12,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:12,642 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,642 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,642 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,643 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:12,643 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:12,643 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,643 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,743 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:12,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:12,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:12,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:12,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:12,746 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:12,746 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,746 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:12,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:12,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,749 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,749 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,750 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,750 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:12,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:12,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:12,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:12,751 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,751 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:12,751 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:12,751 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,753 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:12,755 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:12,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29d27ab9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:12,755 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,756 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:12,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:12,757 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:12,759 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:12,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:12,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:12,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:12,759 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:12,762 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:12,762 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,762 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:12,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,763 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:12,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:12,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:12,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:12,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,766 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:12,766 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:12,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:12,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:12,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:12,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/394dc016a64a4111bd52439c629f8d66 2023-04-16 20:54:12,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/6e9734464824446f825a9c438f308d7b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e9734464824446f825a9c438f308d7b 2023-04-16 20:54:12,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e9734464824446f825a9c438f308d7b, entries=350, sequenceid=117, filesize=52.3 K 2023-04-16 20:54:12,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/24ebcb820b014430aa54d892f7c13fa5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/24ebcb820b014430aa54d892f7c13fa5 2023-04-16 20:54:12,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/24ebcb820b014430aa54d892f7c13fa5, entries=150, sequenceid=117, filesize=11.6 K 2023-04-16 20:54:12,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/394dc016a64a4111bd52439c629f8d66 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/394dc016a64a4111bd52439c629f8d66 2023-04-16 20:54:12,860 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/394dc016a64a4111bd52439c629f8d66, entries=150, sequenceid=117, filesize=11.6 K 2023-04-16 20:54:12,861 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for fb9117f6cdea5099d0a91a955c183b16 in 905ms, sequenceid=117, compaction requested=true 2023-04-16 20:54:12,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:12,861 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:12,861 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:54:12,861 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:12,861 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:54:12,861 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:54:12,864 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 163434 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:54:12,864 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/A is initiating minor compaction (all files) 2023-04-16 20:54:12,864 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/A in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,865 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3e62900126cd48f5a76ffcd761e851d1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e0fa8fe03954f4ca49e521003b8e5df, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e9fe7c707b704b5fb652fc4668f3ff4c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/9dabf7e4a74c4f989fe22d6ec00950cb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e9734464824446f825a9c438f308d7b] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=159.6 K 2023-04-16 20:54:12,865 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:54:12,865 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/B is initiating minor compaction (all files) 2023-04-16 20:54:12,865 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/B in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,865 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/ca264b0ca7f44ea3856041cb8c1587cc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/da5191fd279144ecad1f1c22c59f4fd5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/20bd66316f5f47398652c75f1764c516, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/d30e5455f4604e08bad54e614b9dd957, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/24ebcb820b014430aa54d892f7c13fa5] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=58.0 K 2023-04-16 20:54:12,865 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3e62900126cd48f5a76ffcd761e851d1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681678449993 2023-04-16 20:54:12,865 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ca264b0ca7f44ea3856041cb8c1587cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681678449993 2023-04-16 20:54:12,866 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting da5191fd279144ecad1f1c22c59f4fd5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681678450122 2023-04-16 20:54:12,866 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6e0fa8fe03954f4ca49e521003b8e5df, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681678450122 2023-04-16 20:54:12,866 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:12,866 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 20bd66316f5f47398652c75f1764c516, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681678450444 2023-04-16 20:54:12,866 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e9fe7c707b704b5fb652fc4668f3ff4c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681678450444 2023-04-16 20:54:12,867 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d30e5455f4604e08bad54e614b9dd957, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681678450773 2023-04-16 20:54:12,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:12,867 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9dabf7e4a74c4f989fe22d6ec00950cb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681678450773 2023-04-16 20:54:12,867 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 24ebcb820b014430aa54d892f7c13fa5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681678451944 2023-04-16 20:54:12,868 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6e9734464824446f825a9c438f308d7b, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681678451943 2023-04-16 20:54:12,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:12,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:12,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:12,875 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:12,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:12,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:12,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,877 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:12,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:12,877 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:12,877 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:12,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:12,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:12,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:12,884 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:12,884 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:12,884 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:12,884 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:12,884 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-04-16 20:54:12,884 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:12,884 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:12,885 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:12,885 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:12,885 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:12,885 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:12,891 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#B#compaction#150 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:12,907 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#A#compaction#151 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:13,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:13,129 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:13,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678513136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678513137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678513138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678513139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678513141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678513239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678513241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678513244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,310 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/2424eb54132341c69419f1246b764e21 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2424eb54132341c69419f1246b764e21 2023-04-16 20:54:13,315 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/B of fb9117f6cdea5099d0a91a955c183b16 into 2424eb54132341c69419f1246b764e21(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:13,315 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:13,315 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/B, priority=11, startTime=1681678452861; duration=0sec 2023-04-16 20:54:13,315 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:13,315 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:54:13,317 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:54:13,317 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/C is initiating minor compaction (all files) 2023-04-16 20:54:13,317 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/C in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:13,317 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d7c3dcc9a346491cb1b3d4cef1ea6cfb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/75847f249e1848c1be8ebcb6e718e826, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/38158b5b40564aedbd4813460054f4eb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3fbda78375b84df69958a30ab8d37b9b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/394dc016a64a4111bd52439c629f8d66] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=58.0 K 2023-04-16 20:54:13,317 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d7c3dcc9a346491cb1b3d4cef1ea6cfb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681678449993 2023-04-16 20:54:13,318 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 75847f249e1848c1be8ebcb6e718e826, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681678450122 2023-04-16 20:54:13,318 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 38158b5b40564aedbd4813460054f4eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681678450444 2023-04-16 20:54:13,319 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3fbda78375b84df69958a30ab8d37b9b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681678450773 2023-04-16 20:54:13,319 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 394dc016a64a4111bd52439c629f8d66, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681678451944 2023-04-16 20:54:13,326 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416bc1a9bfe526146759a098a0b09ed21fa to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bc1a9bfe526146759a098a0b09ed21fa 2023-04-16 20:54:13,343 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#C#compaction#153 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:13,362 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/0f154001ddcd48c18d6b88f759d3d7b2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/0f154001ddcd48c18d6b88f759d3d7b2 2023-04-16 20:54:13,366 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/A of fb9117f6cdea5099d0a91a955c183b16 into 0f154001ddcd48c18d6b88f759d3d7b2(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:13,366 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:13,367 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/A, priority=11, startTime=1681678452861; duration=0sec 2023-04-16 20:54:13,367 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:13,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678513442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678513445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678513450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678513745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,750 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/8e0eaa006d3f488faffd1979affde6c2 2023-04-16 20:54:13,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678513749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:13,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678513752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,764 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/3829eeacda8d4029ab0f352426d0e96a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3829eeacda8d4029ab0f352426d0e96a 2023-04-16 20:54:13,765 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/7530be2bd5df49d7a7a82406eca96582 2023-04-16 20:54:13,774 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/C of fb9117f6cdea5099d0a91a955c183b16 into 3829eeacda8d4029ab0f352426d0e96a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:13,774 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:13,774 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/C, priority=11, startTime=1681678452861; duration=0sec 2023-04-16 20:54:13,774 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:13,788 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/19975585ce144395abd8cbc5a110f39d 2023-04-16 20:54:13,793 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/8e0eaa006d3f488faffd1979affde6c2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/8e0eaa006d3f488faffd1979affde6c2 2023-04-16 20:54:13,797 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/8e0eaa006d3f488faffd1979affde6c2, entries=150, sequenceid=143, filesize=25.3 K 2023-04-16 20:54:13,798 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/7530be2bd5df49d7a7a82406eca96582 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7530be2bd5df49d7a7a82406eca96582 2023-04-16 20:54:13,804 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7530be2bd5df49d7a7a82406eca96582, entries=150, sequenceid=143, filesize=11.8 K 2023-04-16 20:54:13,805 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/19975585ce144395abd8cbc5a110f39d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/19975585ce144395abd8cbc5a110f39d 2023-04-16 20:54:13,811 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/19975585ce144395abd8cbc5a110f39d, entries=150, sequenceid=143, filesize=11.8 K 2023-04-16 20:54:13,812 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=46.96 KB/48090 for fb9117f6cdea5099d0a91a955c183b16 in 928ms, sequenceid=143, compaction requested=false 2023-04-16 20:54:13,812 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:13,812 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:13,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:13,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:13,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:13,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:13,814 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,814 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:13,814 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:13,814 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:13,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:13,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@555c1d4c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:13,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:13,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:13,819 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:13,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:13,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:13,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:13,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:13,819 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,819 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:13,820 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:13,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:13,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:13,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:13,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:13,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:13,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:13,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:13,824 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:13,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:13,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:13,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:13,824 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:13,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:13,826 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:13,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:13,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:13,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:13,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:13,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:13,827 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:13,827 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:13,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:13,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,831 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:13,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:13,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:13,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:13,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:13,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:13,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:13,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:13,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:13,832 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:13,832 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:13,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:13,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:13,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:13,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:13,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:13,943 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:13,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:13,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:13,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:13,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:13,947 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:13,947 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:13,947 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:13,947 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:13,948 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:13,948 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:13,950 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,950 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:13,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:13,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:13,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:13,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:13,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:13,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:13,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:13,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:13,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:13,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:13,952 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:13,952 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:13,952 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:54:13,953 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:13,953 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:13,953 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:13,953 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:13,953 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:13,953 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:14,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:14,153 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:14,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681678454170, max timestamp=1681678454180Num uniques 859; 2023-04-16 20:54:14,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 20:54:14,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681678454170, max timestamp=1681678454180Num uniques 859; 2023-04-16 20:54:14,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681678454170, max timestamp=1681678454180Num uniques 859; 2023-04-16 20:54:14,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 20:54:14,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 20:54:14,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678514201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678514201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678514248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678514255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678514260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678514302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678514306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,382 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416f1c79dbec47c4d849c143aa988683b31 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f1c79dbec47c4d849c143aa988683b31 2023-04-16 20:54:14,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678514504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678514514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,788 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=156, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/ecd871861d0640b893dbc7d335d59501 2023-04-16 20:54:14,800 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/96f27d70b7e44c62b13c82976e1e971f 2023-04-16 20:54:14,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678514811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:14,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678514817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,830 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/90bc490b4a284f0e90b2bab0d7a1e754 2023-04-16 20:54:14,835 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/ecd871861d0640b893dbc7d335d59501 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/ecd871861d0640b893dbc7d335d59501 2023-04-16 20:54:14,840 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/ecd871861d0640b893dbc7d335d59501, entries=150, sequenceid=156, filesize=25.3 K 2023-04-16 20:54:14,841 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/96f27d70b7e44c62b13c82976e1e971f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/96f27d70b7e44c62b13c82976e1e971f 2023-04-16 20:54:14,846 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/96f27d70b7e44c62b13c82976e1e971f, entries=150, sequenceid=156, filesize=11.8 K 2023-04-16 20:54:14,849 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/90bc490b4a284f0e90b2bab0d7a1e754 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/90bc490b4a284f0e90b2bab0d7a1e754 2023-04-16 20:54:14,854 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/90bc490b4a284f0e90b2bab0d7a1e754, entries=150, sequenceid=156, filesize=11.8 K 2023-04-16 20:54:14,856 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for fb9117f6cdea5099d0a91a955c183b16 in 904ms, sequenceid=156, compaction requested=true 2023-04-16 20:54:14,856 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:14,857 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:14,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:14,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:14,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:14,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:14,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:14,858 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:14,859 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:14,859 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:14,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:14,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:14,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:14,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1e9b7b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:14,861 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:14,861 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:14,862 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:14,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:14,862 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,862 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:14,864 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:14,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:14,868 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:14,868 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:14,868 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:14,868 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:14,868 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:14,875 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:14,875 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:14,875 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:14,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:14,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:14,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:14,877 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:14,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:14,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:14,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:14,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:14,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:14,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:14,879 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:14,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,879 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:14,879 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:14,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:14,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:14,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:14,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:14,980 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:14,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:14,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:14,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:14,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:14,983 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:14,983 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:14,983 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:14,983 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:14,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:14,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:14,985 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:14,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:14,985 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:14,985 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:14,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:14,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:14,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:14,986 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:14,986 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:14,986 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:14,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:14,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:14,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:14,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:14,987 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-16 20:54:14,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:14,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:14,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:14,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:14,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:14,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:15,006 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304169caa1c385dc3475386bf8338f7d22138 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169caa1c385dc3475386bf8338f7d22138 2023-04-16 20:54:15,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:15,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:15,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678515272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678515273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678515273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678515315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678515323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678515378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678515378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678515378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,413 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=191, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/e55bf89b76ad42a9963bbf3ca55de38b 2023-04-16 20:54:15,424 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/947f492c21bc4fca89e2bef0bea41c98 2023-04-16 20:54:15,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678515582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678515582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678515583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,835 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/0d35ddebce34488bb06e1ca2d8456204 2023-04-16 20:54:15,841 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/e55bf89b76ad42a9963bbf3ca55de38b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e55bf89b76ad42a9963bbf3ca55de38b 2023-04-16 20:54:15,846 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e55bf89b76ad42a9963bbf3ca55de38b, entries=150, sequenceid=191, filesize=25.3 K 2023-04-16 20:54:15,847 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/947f492c21bc4fca89e2bef0bea41c98 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/947f492c21bc4fca89e2bef0bea41c98 2023-04-16 20:54:15,852 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/947f492c21bc4fca89e2bef0bea41c98, entries=150, sequenceid=191, filesize=11.8 K 2023-04-16 20:54:15,853 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/0d35ddebce34488bb06e1ca2d8456204 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d35ddebce34488bb06e1ca2d8456204 2023-04-16 20:54:15,858 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d35ddebce34488bb06e1ca2d8456204, entries=150, sequenceid=191, filesize=11.8 K 2023-04-16 20:54:15,859 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=53.67 KB/54960 for fb9117f6cdea5099d0a91a955c183b16 in 872ms, sequenceid=191, compaction requested=true 2023-04-16 20:54:15,860 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:15,860 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:15,867 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:15,868 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449c0bd[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:15,870 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:15,870 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:15,872 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:15,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:15,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:15,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:15,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:15,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:15,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:15,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:15,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:15,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:15,883 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:15,883 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:15,883 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:15,883 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:15,883 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:15,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:15,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:15,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:15,887 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:15,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:54:15,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:15,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:15,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:15,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:15,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,889 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:15,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:15,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:15,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:15,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:15,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:15,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:15,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:15,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:15,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:15,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:15,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:15,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:15,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:15,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:15,893 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:15,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:15,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:15,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:15,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:15,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:15,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:15,894 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:15,894 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:15,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:15,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678515911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678515916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:15,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678515916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,994 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:15,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:15,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:15,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:15,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:15,997 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:15,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:15,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:15,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:15,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:15,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:15,999 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:15,999 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:15,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,000 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,000 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:16,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:16,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:16,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:16,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:16,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:16,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:16,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,003 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,003 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,003 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:16,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e74753f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:16,004 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:16,004 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,006 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,007 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:16,007 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:16,007 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:16,007 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,007 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:16,008 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:16,008 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,008 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:16,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:16,010 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:16,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:16,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:16,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:16,010 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:16,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,014 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678516017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678516019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678516019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:16,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:16,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,022 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,022 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:16,022 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:16,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,122 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:16,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:16,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:16,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:16,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:16,127 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:16,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:16,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:16,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,133 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,134 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,134 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,134 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:16,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:16,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:16,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:16,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:16,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:16,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:16,135 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,135 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:16,135 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:16,135 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:16,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:16,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:16,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:16,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,137 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,137 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,137 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:16,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf7b9c0[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:16,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:16,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:16,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:16,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:16,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:16,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:16,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:16,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:16,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:16,144 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:16,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:16,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:16,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:16,144 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:16,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,148 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,149 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:16,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:16,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,151 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,151 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:16,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,151 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:16,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678516219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678516222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678516224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,252 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:16,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:16,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:16,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:16,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:16,255 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:16,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:16,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:16,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,257 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,257 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:16,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:16,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:16,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:16,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:16,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:16,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:16,258 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,258 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:16,258 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:16,258 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:16,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:16,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:16,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:16,260 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,261 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,261 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:16,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@655c3e86[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:16,263 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:16,263 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,265 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,265 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:16,265 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:16,265 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:16,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:16,269 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,269 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:16,269 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,269 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:16,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:16,271 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:16,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:16,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:16,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:16,271 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:16,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,273 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,274 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:16,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:16,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:16,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,278 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:16,278 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:16,310 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416254b26fe1a014265bc7fb5ad6a3939a8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416254b26fe1a014265bc7fb5ad6a3939a8 2023-04-16 20:54:16,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678516319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678516331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,378 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:16,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:16,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:16,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:16,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:16,381 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:16,381 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,382 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:16,382 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:16,382 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,382 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,385 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,385 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,385 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,385 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:16,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:16,386 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:16,386 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,386 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:16,386 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:16,386 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:16,386 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,386 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:16,387 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:16,387 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,387 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:16,387 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:16,387 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,387 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:16,387 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:16,388 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,389 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,389 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f5a56b8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:16,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:16,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,392 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,392 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:16,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:16,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:16,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:16,393 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:16,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:16,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:16,396 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:16,397 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:16,397 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:16,397 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:16,397 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:16,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,400 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,401 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,403 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,403 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,403 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,403 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,403 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:16,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:16,403 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,403 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,403 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,403 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,403 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:16,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,403 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:16,403 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,504 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:16,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:16,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:16,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:16,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:16,507 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:16,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:16,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:16,508 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,508 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,509 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:16,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:16,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:16,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:16,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:16,511 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,511 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:16,511 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:16,511 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:16,513 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:16,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321a03e1[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:16,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:16,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,517 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:16,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:16,517 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,517 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:16,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:16,518 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:16,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:16,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:16,522 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:16,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:16,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:16,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:16,522 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:16,523 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,523 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,523 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,526 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678516524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:16,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:16,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,530 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:16,530 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:16,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678516525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,531 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:16,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678516528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,631 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:16,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:16,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:16,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:16,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:16,634 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:16,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:16,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:16,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,637 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:16,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:16,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:16,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:16,639 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,639 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:16,639 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:16,639 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:16,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:16,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:16,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49595db[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:16,643 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:16,643 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,644 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:16,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:16,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:16,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:16,645 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,645 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:16,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,645 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,646 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:16,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:16,648 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:16,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:16,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:16,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:16,648 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:16,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,657 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,658 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,661 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:16,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:16,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,661 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:16,661 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:16,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:16,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,716 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=204, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/2784fb5e5b7d4fff90ff6d5449a5a6c2 2023-04-16 20:54:16,761 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:16,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:16,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:16,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:16,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:16,765 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:16,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:16,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:16,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,767 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,767 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:16,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:16,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:16,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:16,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:16,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:16,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:16,768 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,768 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:16,768 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:16,768 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:16,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:16,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:16,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:16,771 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:16,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b933c1[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:16,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:16,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,774 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:16,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:16,774 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,774 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:16,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:16,776 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:16,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:16,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:16,778 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:16,778 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:16,778 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:16,778 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:16,778 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:16,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,780 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,784 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:16,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:16,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,785 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:16,785 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:16,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:16,885 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:16,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:16,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:16,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:16,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:16,888 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:16,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:16,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:16,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,889 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:16,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:16,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:16,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:16,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:16,891 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,891 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:16,891 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:16,891 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:16,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,892 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:16,892 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:16,893 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,893 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,893 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:16,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dab5af8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:16,895 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:16,895 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,896 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:16,896 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:16,896 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:16,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:16,898 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,898 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:16,898 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,898 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:16,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:16,900 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:16,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:16,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:16,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:16,900 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:16,902 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,902 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:16,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,902 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:16,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,902 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,904 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:16,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:16,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:16,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:16,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:16,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:16,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:16,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:16,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:16,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:16,906 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:16,906 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:16,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:16,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:16,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:16,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:17,006 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:17,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:17,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:17,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:17,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:17,010 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:17,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:17,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:17,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:17,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,011 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,011 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:17,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:17,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:17,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:17,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:17,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:17,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:17,016 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:17,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,017 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:17,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:17,018 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:17,018 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fcc5ba7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:17,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:17,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:17,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:17,019 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,019 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:17,020 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:17,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:17,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:17,024 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:17,025 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:17,025 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:17,025 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:17,025 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:17,026 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:17,026 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,026 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:17,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,026 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,028 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:17,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678517030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:17,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678517033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,036 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:17,036 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:17,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:17,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:17,036 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,036 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:17,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:17,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:17,036 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:17,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:17,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,037 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:17,037 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:17,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:17,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678517035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/6579983e40a24f46abd042f1f38f3842 2023-04-16 20:54:17,137 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:17,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:17,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:17,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:17,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:17,144 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:17,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:17,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:17,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:17,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,146 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:17,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:17,146 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,146 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,146 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:17,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:17,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:17,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:17,148 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:17,148 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:17,148 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:17,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:17,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:17,155 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:17,155 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:17,155 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:17,155 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:17,156 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:17,156 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:17,156 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,156 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:17,156 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:17,158 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:17,159 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:17,159 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,159 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:17,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:17,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:17,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@375f2ec8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:17,162 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,162 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,162 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:17,162 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:17,162 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:17,162 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:17,163 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,163 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:17,163 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:17,163 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:17,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:17,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:17,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:17,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:17,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:17,166 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:17,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:17,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:17,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:17,166 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:17,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:17,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:17,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:17,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:17,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/51893fdaca184000b8d6a86b30153a8a 2023-04-16 20:54:17,168 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,170 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:17,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,173 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,173 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,173 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:17,173 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:17,173 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:17,173 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:17,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:17,173 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:17,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:17,173 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,174 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:17,174 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:17,174 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:17,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:17,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/2784fb5e5b7d4fff90ff6d5449a5a6c2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/2784fb5e5b7d4fff90ff6d5449a5a6c2 2023-04-16 20:54:17,174 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/2784fb5e5b7d4fff90ff6d5449a5a6c2, entries=200, sequenceid=204, filesize=32.1 K 2023-04-16 20:54:17,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/6579983e40a24f46abd042f1f38f3842 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/6579983e40a24f46abd042f1f38f3842 2023-04-16 20:54:17,182 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/6579983e40a24f46abd042f1f38f3842, entries=150, sequenceid=204, filesize=11.8 K 2023-04-16 20:54:17,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/51893fdaca184000b8d6a86b30153a8a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/51893fdaca184000b8d6a86b30153a8a 2023-04-16 20:54:17,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/51893fdaca184000b8d6a86b30153a8a, entries=150, sequenceid=204, filesize=11.8 K 2023-04-16 20:54:17,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for fb9117f6cdea5099d0a91a955c183b16 in 1301ms, sequenceid=204, compaction requested=true 2023-04-16 20:54:17,188 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:17,188 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:17,188 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:54:17,188 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:54:17,188 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:17,188 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:17,190 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136492 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:54:17,190 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/A is initiating minor compaction (all files) 2023-04-16 20:54:17,190 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:54:17,190 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/A in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:17,190 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/B is initiating minor compaction (all files) 2023-04-16 20:54:17,190 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/B in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:17,190 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/0f154001ddcd48c18d6b88f759d3d7b2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/8e0eaa006d3f488faffd1979affde6c2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/ecd871861d0640b893dbc7d335d59501, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e55bf89b76ad42a9963bbf3ca55de38b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/2784fb5e5b7d4fff90ff6d5449a5a6c2] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=133.3 K 2023-04-16 20:54:17,190 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2424eb54132341c69419f1246b764e21, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7530be2bd5df49d7a7a82406eca96582, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/96f27d70b7e44c62b13c82976e1e971f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/947f492c21bc4fca89e2bef0bea41c98, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/6579983e40a24f46abd042f1f38f3842] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=58.8 K 2023-04-16 20:54:17,191 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 0f154001ddcd48c18d6b88f759d3d7b2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681678451944 2023-04-16 20:54:17,191 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 2424eb54132341c69419f1246b764e21, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681678451944 2023-04-16 20:54:17,191 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8e0eaa006d3f488faffd1979affde6c2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681678452008 2023-04-16 20:54:17,191 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7530be2bd5df49d7a7a82406eca96582, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681678452008 2023-04-16 20:54:17,191 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ecd871861d0640b893dbc7d335d59501, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681678453131 2023-04-16 20:54:17,192 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 96f27d70b7e44c62b13c82976e1e971f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681678453131 2023-04-16 20:54:17,192 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e55bf89b76ad42a9963bbf3ca55de38b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681678454194 2023-04-16 20:54:17,192 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 947f492c21bc4fca89e2bef0bea41c98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681678454194 2023-04-16 20:54:17,192 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2784fb5e5b7d4fff90ff6d5449a5a6c2, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681678455262 2023-04-16 20:54:17,193 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6579983e40a24f46abd042f1f38f3842, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681678455262 2023-04-16 20:54:17,216 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#A#compaction#165 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:17,216 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#B#compaction#166 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:17,274 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:17,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:17,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:17,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:17,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:17,277 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:17,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:17,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:17,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:17,278 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,278 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,279 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:17,279 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:17,279 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:17,280 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:17,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:17,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:17,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:17,281 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:17,281 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:17,281 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:17,282 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:17,282 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:17,282 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:17,282 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:17,282 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 20:54:17,282 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:17,282 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:17,282 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:17,282 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:17,282 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:17,282 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:17,638 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/3fba0951dd3441afb3b7dc379c81464d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/3fba0951dd3441afb3b7dc379c81464d 2023-04-16 20:54:17,643 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/B of fb9117f6cdea5099d0a91a955c183b16 into 3fba0951dd3441afb3b7dc379c81464d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:17,643 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:17,643 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/B, priority=11, startTime=1681678457188; duration=0sec 2023-04-16 20:54:17,643 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:17,643 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:54:17,645 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:54:17,645 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/C is initiating minor compaction (all files) 2023-04-16 20:54:17,645 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/C in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:17,645 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3829eeacda8d4029ab0f352426d0e96a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/19975585ce144395abd8cbc5a110f39d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/90bc490b4a284f0e90b2bab0d7a1e754, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d35ddebce34488bb06e1ca2d8456204, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/51893fdaca184000b8d6a86b30153a8a] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=58.8 K 2023-04-16 20:54:17,646 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3829eeacda8d4029ab0f352426d0e96a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681678451944 2023-04-16 20:54:17,646 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 19975585ce144395abd8cbc5a110f39d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681678452008 2023-04-16 20:54:17,646 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 90bc490b4a284f0e90b2bab0d7a1e754, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681678453131 2023-04-16 20:54:17,647 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0d35ddebce34488bb06e1ca2d8456204, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681678454194 2023-04-16 20:54:17,647 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 51893fdaca184000b8d6a86b30153a8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681678455262 2023-04-16 20:54:17,656 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#C#compaction#168 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:17,696 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416a7f04f88032c48a8a99ce5a32e322403 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a7f04f88032c48a8a99ce5a32e322403 2023-04-16 20:54:18,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:18,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:18,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678518051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678518052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678518053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,065 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/d9ce872fb1d44485a8c594606c319e5e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d9ce872fb1d44485a8c594606c319e5e 2023-04-16 20:54:18,071 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/C of fb9117f6cdea5099d0a91a955c183b16 into d9ce872fb1d44485a8c594606c319e5e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:18,071 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:18,071 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/C, priority=11, startTime=1681678457188; duration=0sec 2023-04-16 20:54:18,071 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:18,103 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=224, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/61318119d3694764b5d5cce87fc5565c 2023-04-16 20:54:18,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678518155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678518155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678518157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678518339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,343 DEBUG [Thread-990] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:18,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678518344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,347 DEBUG [Thread-998] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:18,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678518357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678518358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678518359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,452 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/1341abcbb3264a258bc140ee7a87f4b0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1341abcbb3264a258bc140ee7a87f4b0 2023-04-16 20:54:18,457 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/A of fb9117f6cdea5099d0a91a955c183b16 into 1341abcbb3264a258bc140ee7a87f4b0(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-16 20:54:18,457 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:18,457 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/A, priority=11, startTime=1681678457188; duration=1sec 2023-04-16 20:54:18,457 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:18,514 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/4f40b57b0e9f49ec9e4d0a4306411de1 2023-04-16 20:54:18,528 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/e4e44be0efbf4350b8a541d11bd21360 2023-04-16 20:54:18,536 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/61318119d3694764b5d5cce87fc5565c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/61318119d3694764b5d5cce87fc5565c 2023-04-16 20:54:18,541 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/61318119d3694764b5d5cce87fc5565c, entries=150, sequenceid=224, filesize=25.3 K 2023-04-16 20:54:18,541 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/4f40b57b0e9f49ec9e4d0a4306411de1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/4f40b57b0e9f49ec9e4d0a4306411de1 2023-04-16 20:54:18,555 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/4f40b57b0e9f49ec9e4d0a4306411de1, entries=150, sequenceid=224, filesize=11.8 K 2023-04-16 20:54:18,556 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/e4e44be0efbf4350b8a541d11bd21360 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/e4e44be0efbf4350b8a541d11bd21360 2023-04-16 20:54:18,560 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/e4e44be0efbf4350b8a541d11bd21360, entries=150, sequenceid=224, filesize=11.8 K 2023-04-16 20:54:18,561 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fb9117f6cdea5099d0a91a955c183b16 in 1279ms, sequenceid=224, compaction requested=false 2023-04-16 20:54:18,561 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:18,561 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:18,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:18,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e3124d9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:18,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:18,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,567 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,568 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,568 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:18,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:18,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:18,578 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:18,578 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:18,578 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:18,578 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:18,578 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:18,580 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,580 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:18,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,580 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,582 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,587 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,587 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,587 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,587 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:18,587 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:18,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:18,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:18,587 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,587 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,587 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,587 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:18,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:18,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:18,587 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:18,588 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:18,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:18,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:18,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:18,661 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:54:18,662 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:18,662 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:18,662 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:18,662 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:18,662 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:18,662 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:18,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678518682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678518683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678518683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,688 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:18,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:18,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:18,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:18,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:18,691 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:18,691 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:18,691 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:18,691 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:18,692 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,692 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,693 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,693 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:18,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:18,693 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,693 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:18,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:18,695 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:18,695 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:18,695 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:18,695 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:18,698 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:18,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@596add7d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:18,700 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:18,700 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,705 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,705 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:18,705 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,705 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:18,705 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:18,705 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:18,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:18,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:18,706 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:18,709 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:18,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:18,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:18,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:18,709 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:18,710 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,710 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,710 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:18,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,712 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:18,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,718 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:18,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:18,718 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,719 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:18,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:18,719 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,719 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,719 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:18,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678518784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678518786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678518787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,819 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:18,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:18,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:18,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:18,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:18,822 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,823 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:18,823 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:18,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:18,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:18,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:18,825 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:18,825 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:18,825 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:18,825 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:18,827 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:18,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2406b494[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:18,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:18,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:18,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:18,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:18,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:18,834 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:18,834 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:18,834 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:18,834 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:18,834 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:18,835 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,835 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,835 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,836 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:18,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:18,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:18,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:18,839 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:18,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:18,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:18,839 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:18,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:18,840 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:18,940 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:18,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:18,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:18,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:18,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:18,943 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:18,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:18,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:18,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:18,944 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,944 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,945 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:18,945 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:18,945 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,945 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:18,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:18,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:18,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:18,946 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:18,946 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:18,946 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:18,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:18,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:18,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:18,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:18,949 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:18,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a61e7b2[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:18,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:18,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,952 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:18,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:18,953 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:18,956 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:18,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:18,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:18,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:18,956 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:18,957 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,957 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,957 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:18,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,958 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:18,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:18,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:18,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:18,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:18,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:18,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:18,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:18,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:18,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:18,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:18,963 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:18,963 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:18,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:18,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:18,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678518987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678518989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:18,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:18,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678518991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,063 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:19,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:19,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:19,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:19,066 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:19,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:19,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:19,067 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,067 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,068 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,068 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:19,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:19,068 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,068 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:19,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:19,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:19,069 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:19,069 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,069 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:19,070 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,070 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:19,070 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:19,070 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:19,072 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,074 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:19,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:19,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d0101cb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:19,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,077 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,077 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:19,077 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:19,077 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:19,078 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,078 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:19,079 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304161b0c0ef26519485fbbe5e9bff07103b5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161b0c0ef26519485fbbe5e9bff07103b5 2023-04-16 20:54:19,080 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:19,080 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,080 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:19,080 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:19,082 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:19,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:19,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:19,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:19,082 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:19,084 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,084 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,084 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:19,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,086 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:19,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:19,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:19,088 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:19,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:19,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:19,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:19,088 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:19,088 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:19,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:19,189 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:19,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:19,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:19,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:19,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:19,192 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:19,192 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,192 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:19,192 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:19,193 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,193 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,194 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:19,194 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:19,194 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,194 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:19,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:19,195 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,195 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:19,196 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:19,196 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:19,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:19,200 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,200 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:19,202 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:19,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@120fa623[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:19,202 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,203 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:19,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:19,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:19,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:19,204 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,204 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,205 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,205 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:19,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:19,207 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:19,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:19,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:19,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:19,207 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:19,209 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,209 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,209 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:19,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,210 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,211 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:19,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:19,213 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:19,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:19,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:19,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:19,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:19,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,213 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:19,213 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:19,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:19,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:19,214 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:19,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678519289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:19,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678519290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:19,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678519294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,313 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:19,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:19,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:19,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:19,316 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:19,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:19,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:19,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,318 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:19,318 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:19,318 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,318 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:19,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:19,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:19,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:19,320 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,320 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:19,320 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:19,320 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:19,320 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:19,321 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,321 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,321 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,321 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:19,323 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:19,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c12e2[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:19,323 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:19,325 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,325 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:19,326 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:19,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:19,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:19,328 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:19,328 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:19,328 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:19,328 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:19,328 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:19,330 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,330 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,330 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:19,330 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,331 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:19,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:19,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:19,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:19,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:19,334 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:19,334 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:19,334 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,334 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:19,334 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:19,434 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:19,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:19,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:19,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:19,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:19,437 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:19,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:19,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:19,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:19,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:19,439 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:19,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:19,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:19,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:19,441 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,441 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:19,441 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:19,441 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:19,443 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:19,444 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66852940[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:19,444 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:19,446 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,446 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:19,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:19,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:19,447 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:19,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:19,449 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:19,449 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:19,449 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:19,449 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:19,449 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:19,454 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,454 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,454 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:19,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:19,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,455 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:19,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:19,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:19,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:19,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,458 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:19,458 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:19,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:19,458 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:19,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:19,458 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:19,458 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:19,498 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/4de6de21c91a4b1b807f268d87724ae8 2023-04-16 20:54:19,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/67a9ddaf5fd84d25bde3ae342d5c95fc 2023-04-16 20:54:19,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/fff9cab247c1400691f2484b17f1eb55 2023-04-16 20:54:19,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/4de6de21c91a4b1b807f268d87724ae8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/4de6de21c91a4b1b807f268d87724ae8 2023-04-16 20:54:19,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/4de6de21c91a4b1b807f268d87724ae8, entries=200, sequenceid=240, filesize=32.1 K 2023-04-16 20:54:19,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/67a9ddaf5fd84d25bde3ae342d5c95fc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/67a9ddaf5fd84d25bde3ae342d5c95fc 2023-04-16 20:54:19,558 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:19,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:19,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:19,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/67a9ddaf5fd84d25bde3ae342d5c95fc, entries=150, sequenceid=240, filesize=11.8 K 2023-04-16 20:54:19,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/fff9cab247c1400691f2484b17f1eb55 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fff9cab247c1400691f2484b17f1eb55 2023-04-16 20:54:19,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:19,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:19,564 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:19,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:19,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:19,565 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,565 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:19,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:19,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fff9cab247c1400691f2484b17f1eb55, entries=150, sequenceid=240, filesize=11.8 K 2023-04-16 20:54:19,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:19,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for fb9117f6cdea5099d0a91a955c183b16 in 905ms, sequenceid=240, compaction requested=true 2023-04-16 20:54:19,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:19,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:19,567 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:19,567 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:19,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:19,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:19,568 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:19,568 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/A is initiating minor compaction (all files) 2023-04-16 20:54:19,568 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/A in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,568 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1341abcbb3264a258bc140ee7a87f4b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/61318119d3694764b5d5cce87fc5565c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/4de6de21c91a4b1b807f268d87724ae8] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=83.0 K 2023-04-16 20:54:19,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:19,568 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:19,568 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/B is initiating minor compaction (all files) 2023-04-16 20:54:19,569 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1341abcbb3264a258bc140ee7a87f4b0, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681678455262 2023-04-16 20:54:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:19,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:19,569 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/B in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,569 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/3fba0951dd3441afb3b7dc379c81464d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/4f40b57b0e9f49ec9e4d0a4306411de1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/67a9ddaf5fd84d25bde3ae342d5c95fc] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=35.6 K 2023-04-16 20:54:19,569 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 61318119d3694764b5d5cce87fc5565c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1681678455909 2023-04-16 20:54:19,569 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3fba0951dd3441afb3b7dc379c81464d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681678455262 2023-04-16 20:54:19,569 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 4de6de21c91a4b1b807f268d87724ae8, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681678458049 2023-04-16 20:54:19,570 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 4f40b57b0e9f49ec9e4d0a4306411de1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1681678455909 2023-04-16 20:54:19,570 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 67a9ddaf5fd84d25bde3ae342d5c95fc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681678458049 2023-04-16 20:54:19,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:19,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:19,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:19,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:19,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:19,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:19,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:19,571 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,571 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:19,571 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-16 20:54:19,572 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:19,572 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:19,572 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:19,572 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:19,572 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:19,572 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:19,590 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#B#compaction#176 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:19,598 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#A#compaction#174 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:19,674 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/c5391745c14c4758aa5223664492d1e1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/c5391745c14c4758aa5223664492d1e1 2023-04-16 20:54:19,679 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/A of fb9117f6cdea5099d0a91a955c183b16 into c5391745c14c4758aa5223664492d1e1(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:19,679 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:19,679 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/A, priority=13, startTime=1681678459567; duration=0sec 2023-04-16 20:54:19,680 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:19,680 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:19,681 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:19,681 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/C is initiating minor compaction (all files) 2023-04-16 20:54:19,682 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/C in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:19,682 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d9ce872fb1d44485a8c594606c319e5e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/e4e44be0efbf4350b8a541d11bd21360, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fff9cab247c1400691f2484b17f1eb55] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=35.6 K 2023-04-16 20:54:19,682 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d9ce872fb1d44485a8c594606c319e5e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681678455262 2023-04-16 20:54:19,682 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e4e44be0efbf4350b8a541d11bd21360, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1681678455909 2023-04-16 20:54:19,683 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting fff9cab247c1400691f2484b17f1eb55, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681678458049 2023-04-16 20:54:19,689 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#C#compaction#177 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:19,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:19,794 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:19,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678519804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678519804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678519806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678519909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678519910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678519910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:20,011 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/8e2999c87bf344439c1224d208350621 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8e2999c87bf344439c1224d208350621 2023-04-16 20:54:20,015 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041684088be85b9540c791d4d150b39f3715 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041684088be85b9540c791d4d150b39f3715 2023-04-16 20:54:20,017 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/B of fb9117f6cdea5099d0a91a955c183b16 into 8e2999c87bf344439c1224d208350621(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:20,017 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:20,017 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/B, priority=13, startTime=1681678459567; duration=0sec 2023-04-16 20:54:20,017 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:20,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:20,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678520113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:20,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:20,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678520113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:20,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:20,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678520113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:20,133 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/b469c4950a1445d79fb1a12986913b3d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/b469c4950a1445d79fb1a12986913b3d 2023-04-16 20:54:20,139 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/C of fb9117f6cdea5099d0a91a955c183b16 into b469c4950a1445d79fb1a12986913b3d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:20,139 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:20,140 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/C, priority=13, startTime=1681678459567; duration=0sec 2023-04-16 20:54:20,140 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:20,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:20,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678520416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:20,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:20,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678520417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:20,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:20,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678520418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:20,422 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/3b3a7f497130477fa3a51e32f0da4c39 2023-04-16 20:54:20,832 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/e4c6057e9e44490b976ba91ec15a6766 2023-04-16 20:54:20,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:20,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678520918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:20,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:20,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59886 deadline: 1681678520921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:20,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:20,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59922 deadline: 1681678520922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,253 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/306d888feb7c4725bca04e1e389842bc 2023-04-16 20:54:21,258 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/3b3a7f497130477fa3a51e32f0da4c39 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3b3a7f497130477fa3a51e32f0da4c39 2023-04-16 20:54:21,262 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3b3a7f497130477fa3a51e32f0da4c39, entries=150, sequenceid=259, filesize=25.4 K 2023-04-16 20:54:21,264 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/e4c6057e9e44490b976ba91ec15a6766 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e4c6057e9e44490b976ba91ec15a6766 2023-04-16 20:54:21,267 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e4c6057e9e44490b976ba91ec15a6766, entries=150, sequenceid=259, filesize=11.9 K 2023-04-16 20:54:21,268 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/306d888feb7c4725bca04e1e389842bc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/306d888feb7c4725bca04e1e389842bc 2023-04-16 20:54:21,273 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/306d888feb7c4725bca04e1e389842bc, entries=150, sequenceid=259, filesize=11.9 K 2023-04-16 20:54:21,273 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for fb9117f6cdea5099d0a91a955c183b16 in 1702ms, sequenceid=259, compaction requested=false 2023-04-16 20:54:21,273 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:21,273 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:21,273 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:21,273 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:21,273 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,274 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:21,274 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:21,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,275 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:21,275 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:21,275 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:21,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:21,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731f7302[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:21,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:21,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:21,281 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:21,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:21,282 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:21,282 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:21,282 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:21,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:21,283 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:21,283 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,283 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:21,283 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:21,285 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:21,285 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:21,285 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:21,285 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:21,285 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:21,287 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:21,287 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:21,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:21,287 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:21,287 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:21,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:21,289 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:21,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:21,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:21,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:21,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:21,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:21,291 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:21,291 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:21,291 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:21,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:21,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:21,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:21,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:21,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:21,292 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:21,391 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:21,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:21,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:21,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:21,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:21,395 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:21,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:21,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:21,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:21,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:21,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:21,398 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:21,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:21,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:21,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:21,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:21,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:21,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:21,399 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:21,399 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:21,399 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:54:21,400 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:21,400 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:21,400 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:21,400 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:21,400 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:21,400 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:21,816 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416cca6a56fc5e04e2d9d0309d7ec1bf688 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416cca6a56fc5e04e2d9d0309d7ec1bf688 2023-04-16 20:54:21,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:21,927 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:21,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:21,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678521942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:21,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59886 deadline: 1681678521945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:21,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:21,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59922 deadline: 1681678521945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59884 deadline: 1681678522046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59886 deadline: 1681678522046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59922 deadline: 1681678522048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,229 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/1fa4a5acd93c468ca67389849e09275a 2023-04-16 20:54:22,247 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/b27fb05c301443b582627be3e2e9dd68 2023-04-16 20:54:22,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59884 deadline: 1681678522248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59886 deadline: 1681678522249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59922 deadline: 1681678522249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,259 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/02a2e3fd1f604515a99d9408455e32f4 2023-04-16 20:54:22,263 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/1fa4a5acd93c468ca67389849e09275a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1fa4a5acd93c468ca67389849e09275a 2023-04-16 20:54:22,267 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1fa4a5acd93c468ca67389849e09275a, entries=150, sequenceid=275, filesize=25.4 K 2023-04-16 20:54:22,268 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/b27fb05c301443b582627be3e2e9dd68 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/b27fb05c301443b582627be3e2e9dd68 2023-04-16 20:54:22,272 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/b27fb05c301443b582627be3e2e9dd68, entries=150, sequenceid=275, filesize=11.9 K 2023-04-16 20:54:22,273 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/02a2e3fd1f604515a99d9408455e32f4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/02a2e3fd1f604515a99d9408455e32f4 2023-04-16 20:54:22,284 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/02a2e3fd1f604515a99d9408455e32f4, entries=150, sequenceid=275, filesize=11.9 K 2023-04-16 20:54:22,285 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for fb9117f6cdea5099d0a91a955c183b16 in 886ms, sequenceid=275, compaction requested=true 2023-04-16 20:54:22,285 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:22,285 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:22,288 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:22,289 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,290 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,290 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:22,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c4eedfc[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:22,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:22,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,293 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,293 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:22,293 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:22,293 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:22,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:22,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:22,295 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:22,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:22,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:22,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:22,295 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:22,298 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,299 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,299 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,299 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,304 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:22,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:22,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,304 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,304 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,304 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,305 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:22,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,305 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:22,305 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,305 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,305 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,305 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,305 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:22,357 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 20:54:22,357 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:22,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:22,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:22,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:22,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:22,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:22,379 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416b123b4e660ba48aca06165153bd3aacd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b123b4e660ba48aca06165153bd3aacd 2023-04-16 20:54:22,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678522389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678522389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,405 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:22,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:22,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:22,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:22,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:22,409 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:22,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:22,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:22,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,411 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,411 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:22,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:22,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:22,413 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,413 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:22,413 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:22,413 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:22,414 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:22,414 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,414 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:22,414 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:22,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:22,417 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:22,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f865795[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:22,417 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,419 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:22,419 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,419 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:22,420 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:22,424 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:22,424 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:22,424 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:22,424 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:22,424 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:22,426 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,426 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,426 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,427 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:22,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:22,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,430 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:22,430 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:22,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678522495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678522495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,531 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:22,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:22,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:22,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:22,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:22,534 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:22,534 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,534 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:22,534 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:22,534 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,534 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,537 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:22,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:22,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:22,539 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,539 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:22,539 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:22,539 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:22,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,541 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3006ae0c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:22,543 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:22,543 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:22,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:22,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:22,544 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,544 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:22,545 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,545 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:22,545 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,545 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:22,548 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:22,548 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:22,548 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:22,548 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:22,548 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:22,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,553 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,553 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,554 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59886 deadline: 1681678522551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59884 deadline: 1681678522552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,554 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59922 deadline: 1681678522552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,556 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,558 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,558 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,559 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:22,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:22,559 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,559 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,559 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,559 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,559 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,559 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,559 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,559 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:22,559 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:22,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:22,659 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:22,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:22,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:22,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:22,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:22,663 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:22,663 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,663 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:22,663 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:22,663 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,663 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,664 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,665 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,665 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:22,665 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:22,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:22,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:22,667 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,667 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:22,667 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:22,667 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:22,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:22,668 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,669 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,669 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,669 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38895c6d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:22,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:22,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,673 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:22,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:22,673 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,673 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:22,674 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:22,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:22,676 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:22,677 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:22,677 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:22,677 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:22,677 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:22,678 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,678 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,678 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,678 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,680 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,682 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,682 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,682 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,682 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,682 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,682 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:22,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:22,682 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,682 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,683 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:22,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,683 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:22,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678522697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678522698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,783 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:22,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:22,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:22,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:22,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:22,786 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:22,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:22,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:22,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,789 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:22,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:22,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:22,792 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=295, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/73d93ecd56d340e784f21d582aa6f2e3 2023-04-16 20:54:22,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:22,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:22,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:22,793 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,793 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:22,793 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:22,793 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,794 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,795 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:22,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5343678d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:22,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:22,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,798 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,798 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:22,798 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:22,798 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:22,800 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:22,800 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,800 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:22,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:22,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:22,803 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:22,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:22,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:22,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:22,803 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:22,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/7a411c377eed478a935bbd9a7e3e2d40 2023-04-16 20:54:22,806 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,807 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,807 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,812 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:22,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:22,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,813 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:22,813 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:22,913 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:22,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:22,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:22,917 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:22,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:22,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:22,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,919 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:22,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:22,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:22,922 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,922 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:22,922 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:22,922 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,924 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:22,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb4a88f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:22,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:22,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,927 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:22,927 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:22,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:22,927 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:22,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:22,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:22,931 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:22,931 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:22,931 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:22,931 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:22,931 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:22,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:22,933 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:22,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,933 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,934 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:22,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:22,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,940 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:22,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:22,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:22,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:22,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:22,941 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:22,941 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:22,941 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:22,941 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:22,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:22,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:22,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:22,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678522998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678523001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,041 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:23,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:23,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:23,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:23,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:23,044 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:23,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:23,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:23,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:23,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,045 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,045 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:23,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:23,046 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,046 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:23,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:23,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:23,047 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:23,047 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:23,047 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:23,047 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:23,047 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:23,047 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,047 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:23,047 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:23,047 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,047 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:23,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:23,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:23,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:23,049 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,049 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,049 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,049 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:23,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b47039b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:23,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:23,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,052 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,052 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:23,052 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:23,052 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:23,052 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,052 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:23,055 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:23,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:23,057 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:23,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:23,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:23,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:23,057 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:23,059 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,059 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,059 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,059 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59884 deadline: 1681678523056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:23,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59922 deadline: 1681678523056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,061 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59886 deadline: 1681678523058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:23,063 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:23,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:23,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:23,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:23,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:23,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:23,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:23,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:23,063 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:23,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,063 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:23,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:23,163 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:23,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:23,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:23,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:23,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:23,166 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:23,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:23,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:23,166 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:23,167 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,167 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,168 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:23,168 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:23,168 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,168 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:23,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:23,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:23,169 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:23,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:23,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:23,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:23,172 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,172 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,172 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,173 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:23,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:23,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69df5eaa[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:23,175 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:23,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:23,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:23,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:23,176 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:23,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:23,179 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:23,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:23,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:23,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:23,179 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:23,180 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,180 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,180 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:23,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,182 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:23,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:23,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:23,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:23,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:23,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:23,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,187 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,187 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,187 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:23,187 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:23,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:23,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/ca8bc94269994bf7a0153383df627143 2023-04-16 20:54:23,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/73d93ecd56d340e784f21d582aa6f2e3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/73d93ecd56d340e784f21d582aa6f2e3 2023-04-16 20:54:23,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/73d93ecd56d340e784f21d582aa6f2e3, entries=200, sequenceid=295, filesize=32.3 K 2023-04-16 20:54:23,228 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/7a411c377eed478a935bbd9a7e3e2d40 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7a411c377eed478a935bbd9a7e3e2d40 2023-04-16 20:54:23,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7a411c377eed478a935bbd9a7e3e2d40, entries=150, sequenceid=295, filesize=11.9 K 2023-04-16 20:54:23,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/ca8bc94269994bf7a0153383df627143 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ca8bc94269994bf7a0153383df627143 2023-04-16 20:54:23,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ca8bc94269994bf7a0153383df627143, entries=150, sequenceid=295, filesize=11.9 K 2023-04-16 20:54:23,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fb9117f6cdea5099d0a91a955c183b16 in 885ms, sequenceid=295, compaction requested=true 2023-04-16 20:54:23,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:23,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:23,242 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:23,242 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:23,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:23,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:23,244 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48988 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:23,244 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/B is initiating minor compaction (all files) 2023-04-16 20:54:23,244 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111468 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:23,244 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/B in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,244 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/A is initiating minor compaction (all files) 2023-04-16 20:54:23,244 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8e2999c87bf344439c1224d208350621, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e4c6057e9e44490b976ba91ec15a6766, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/b27fb05c301443b582627be3e2e9dd68, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7a411c377eed478a935bbd9a7e3e2d40] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=47.8 K 2023-04-16 20:54:23,244 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/A in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,244 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/c5391745c14c4758aa5223664492d1e1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3b3a7f497130477fa3a51e32f0da4c39, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1fa4a5acd93c468ca67389849e09275a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/73d93ecd56d340e784f21d582aa6f2e3] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=108.9 K 2023-04-16 20:54:23,244 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8e2999c87bf344439c1224d208350621, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681678458049 2023-04-16 20:54:23,244 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c5391745c14c4758aa5223664492d1e1, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681678458049 2023-04-16 20:54:23,245 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e4c6057e9e44490b976ba91ec15a6766, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681678458676 2023-04-16 20:54:23,245 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3b3a7f497130477fa3a51e32f0da4c39, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681678458676 2023-04-16 20:54:23,245 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b27fb05c301443b582627be3e2e9dd68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681678459801 2023-04-16 20:54:23,246 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1fa4a5acd93c468ca67389849e09275a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681678459801 2023-04-16 20:54:23,246 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7a411c377eed478a935bbd9a7e3e2d40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1681678461937 2023-04-16 20:54:23,246 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 73d93ecd56d340e784f21d582aa6f2e3, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1681678461937 2023-04-16 20:54:23,261 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#B#compaction#187 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:23,264 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#A#compaction#186 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:23,272 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/e9c21d6ab96c41f09a7dac68110b1611 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e9c21d6ab96c41f09a7dac68110b1611 2023-04-16 20:54:23,283 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/B of fb9117f6cdea5099d0a91a955c183b16 into e9c21d6ab96c41f09a7dac68110b1611(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:23,283 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:23,283 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/B, priority=12, startTime=1681678463242; duration=0sec 2023-04-16 20:54:23,283 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:23,283 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:23,285 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48988 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:23,285 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/C is initiating minor compaction (all files) 2023-04-16 20:54:23,285 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/C in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,285 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/b469c4950a1445d79fb1a12986913b3d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/306d888feb7c4725bca04e1e389842bc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/02a2e3fd1f604515a99d9408455e32f4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ca8bc94269994bf7a0153383df627143] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=47.8 K 2023-04-16 20:54:23,286 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b469c4950a1445d79fb1a12986913b3d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681678458049 2023-04-16 20:54:23,286 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 306d888feb7c4725bca04e1e389842bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681678458676 2023-04-16 20:54:23,286 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 02a2e3fd1f604515a99d9408455e32f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681678459801 2023-04-16 20:54:23,287 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:23,287 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ca8bc94269994bf7a0153383df627143, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1681678461937 2023-04-16 20:54:23,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:23,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:23,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:23,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:23,292 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:23,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:23,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:23,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:23,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,294 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:23,294 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:23,294 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:23,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:23,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:23,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:23,297 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:23,297 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:23,297 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:23,308 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:23,308 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:23,308 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,308 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:23,308 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:54:23,308 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:23,308 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:23,309 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:23,309 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:23,309 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:23,309 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:23,318 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#C#compaction#188 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:23,349 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416278a8cee54ac458b89bf0a12ca783ab7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416278a8cee54ac458b89bf0a12ca783ab7 2023-04-16 20:54:23,355 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=308, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/1d8c7eb2ecb048218693dff95bef8280 2023-04-16 20:54:23,382 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/097d8ef7a80d45079a98cf4871a69376 2023-04-16 20:54:23,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:23,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:23,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678523527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678523528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678523630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678523631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,731 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/a4936ab52985420aaead9bc00501cb78 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/a4936ab52985420aaead9bc00501cb78 2023-04-16 20:54:23,736 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/A of fb9117f6cdea5099d0a91a955c183b16 into a4936ab52985420aaead9bc00501cb78(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:23,736 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:23,736 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/A, priority=12, startTime=1681678463242; duration=0sec 2023-04-16 20:54:23,736 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:23,738 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/ef98a3dd3ea845e7bce75d140098d5cb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ef98a3dd3ea845e7bce75d140098d5cb 2023-04-16 20:54:23,743 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/C of fb9117f6cdea5099d0a91a955c183b16 into ef98a3dd3ea845e7bce75d140098d5cb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:23,743 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:23,743 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/C, priority=12, startTime=1681678463242; duration=0sec 2023-04-16 20:54:23,743 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:23,800 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/55b838c1e0ec4b348e2e3b73d675ae94 2023-04-16 20:54:23,805 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/1d8c7eb2ecb048218693dff95bef8280 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1d8c7eb2ecb048218693dff95bef8280 2023-04-16 20:54:23,810 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1d8c7eb2ecb048218693dff95bef8280, entries=150, sequenceid=308, filesize=25.4 K 2023-04-16 20:54:23,811 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/097d8ef7a80d45079a98cf4871a69376 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/097d8ef7a80d45079a98cf4871a69376 2023-04-16 20:54:23,816 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/097d8ef7a80d45079a98cf4871a69376, entries=150, sequenceid=308, filesize=11.9 K 2023-04-16 20:54:23,817 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/55b838c1e0ec4b348e2e3b73d675ae94 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/55b838c1e0ec4b348e2e3b73d675ae94 2023-04-16 20:54:23,821 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/55b838c1e0ec4b348e2e3b73d675ae94, entries=150, sequenceid=308, filesize=11.9 K 2023-04-16 20:54:23,822 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fb9117f6cdea5099d0a91a955c183b16 in 514ms, sequenceid=308, compaction requested=false 2023-04-16 20:54:23,822 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:23,822 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:23,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,826 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,826 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:23,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21e83313[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:23,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:23,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:23,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:23,833 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:23,833 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:23,833 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,833 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:23,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-16 20:54:23,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:23,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:23,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:23,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:23,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:23,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:23,838 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:23,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:23,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:23,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:23,838 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:23,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:23,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:23,841 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,842 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,847 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:23,847 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:23,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:23,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:23,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:23,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:23,848 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:23,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,849 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:23,849 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:23,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:23,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678523850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678523852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,869 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416f1a8b89f593641df90382ca4fbd9370e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f1a8b89f593641df90382ca4fbd9370e 2023-04-16 20:54:23,874 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=331, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/32e53ee2d554405e87784e78585899c3 2023-04-16 20:54:23,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/8250cb43a89d42bf91d03dd098427a79 2023-04-16 20:54:23,902 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/29d6e1861d4f403da60f4cfffa6f0a32 2023-04-16 20:54:23,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/32e53ee2d554405e87784e78585899c3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/32e53ee2d554405e87784e78585899c3 2023-04-16 20:54:23,910 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/32e53ee2d554405e87784e78585899c3, entries=200, sequenceid=331, filesize=32.3 K 2023-04-16 20:54:23,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/8250cb43a89d42bf91d03dd098427a79 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8250cb43a89d42bf91d03dd098427a79 2023-04-16 20:54:23,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8250cb43a89d42bf91d03dd098427a79, entries=150, sequenceid=331, filesize=11.9 K 2023-04-16 20:54:23,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/29d6e1861d4f403da60f4cfffa6f0a32 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/29d6e1861d4f403da60f4cfffa6f0a32 2023-04-16 20:54:23,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/29d6e1861d4f403da60f4cfffa6f0a32, entries=150, sequenceid=331, filesize=11.9 K 2023-04-16 20:54:23,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for fb9117f6cdea5099d0a91a955c183b16 in 85ms, sequenceid=331, compaction requested=true 2023-04-16 20:54:23,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:23,921 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:23,921 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:23,921 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:23,921 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:23,921 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:23,922 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:23,923 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:23,923 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/A is initiating minor compaction (all files) 2023-04-16 20:54:23,923 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/B is initiating minor compaction (all files) 2023-04-16 20:54:23,923 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/A in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,923 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/B in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,923 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/a4936ab52985420aaead9bc00501cb78, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1d8c7eb2ecb048218693dff95bef8280, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/32e53ee2d554405e87784e78585899c3] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=83.7 K 2023-04-16 20:54:23,923 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e9c21d6ab96c41f09a7dac68110b1611, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/097d8ef7a80d45079a98cf4871a69376, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8250cb43a89d42bf91d03dd098427a79] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=36.3 K 2023-04-16 20:54:23,923 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting a4936ab52985420aaead9bc00501cb78, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1681678461937 2023-04-16 20:54:23,923 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e9c21d6ab96c41f09a7dac68110b1611, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1681678461937 2023-04-16 20:54:23,924 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1d8c7eb2ecb048218693dff95bef8280, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681678462362 2023-04-16 20:54:23,925 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 097d8ef7a80d45079a98cf4871a69376, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681678462362 2023-04-16 20:54:23,925 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 32e53ee2d554405e87784e78585899c3, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681678463521 2023-04-16 20:54:23,925 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8250cb43a89d42bf91d03dd098427a79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681678463526 2023-04-16 20:54:23,948 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#B#compaction#196 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:23,949 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:23,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:23,951 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#A#compaction#195 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:23,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:23,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:23,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:54:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:23,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:23,955 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:23,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:23,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:23,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:23,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:23,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:23,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:23,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:23,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:23,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,958 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:23,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:23,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:23,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:23,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:23,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:23,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:23,964 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,964 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:23,964 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:23,964 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,968 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:23,968 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:23,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,969 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:23,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a230771[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:23,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:23,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,973 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,973 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:23,973 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:23,973 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:23,973 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,973 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:23,974 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:23,978 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:23,978 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:23,978 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:23,978 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:23,978 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:23,981 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,981 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,981 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:23,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:23,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678523982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59906 deadline: 1681678523983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:23,987 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304161bc40d818b2d4814b7495369154d7171 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161bc40d818b2d4814b7495369154d7171 2023-04-16 20:54:23,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:23,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:23,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:23,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:23,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:23,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:23,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:23,989 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:23,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:23,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:23,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:23,990 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:23,990 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:23,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:23,998 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=343, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/68363b51d36844b4bd1811b193829152 2023-04-16 20:54:24,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/8af48a7e7eb449bda7dd5ecaca4b6592 2023-04-16 20:54:24,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59884 deadline: 1681678524066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59922 deadline: 1681678524068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59886 deadline: 1681678524071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678524087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,090 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:24,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:24,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678524088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:24,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:24,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:24,093 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:24,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:24,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:24,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:24,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:24,094 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:24,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:24,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:24,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:24,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:24,096 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:24,096 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:24,096 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:24,096 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:24,098 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:24,099 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395b9475[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:24,100 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,102 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:24,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:24,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:24,102 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,102 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:24,103 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:24,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:24,105 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:24,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:24,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:24,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:24,106 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:24,107 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,107 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,107 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,108 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,110 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:24,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:24,112 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:24,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:24,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,112 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:24,112 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:24,113 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:24,113 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:24,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:24,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:24,213 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:24,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:24,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:24,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:24,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:24,216 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:24,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:24,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:24,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:24,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:24,223 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:24,223 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,223 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:24,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:24,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:24,225 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:24,225 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:24,225 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:24,225 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:24,226 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,227 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,227 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:24,228 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:24,228 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a62a7fa[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:24,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,229 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:24,229 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,229 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:24,229 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:24,229 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:24,231 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:24,233 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:24,233 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:24,233 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:24,233 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:24,233 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:24,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:24,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:24,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,237 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:24,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,239 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:24,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:24,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:24,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:24,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:24,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:24,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,240 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:24,240 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,240 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:24,240 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,240 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,240 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,240 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:24,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59890 deadline: 1681678524291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678524293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,340 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:24,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:24,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:24,343 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:24,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:24,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:24,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:24,344 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,344 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:24,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:24,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:24,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:24,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:24,347 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:24,347 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:24,347 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:24,347 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:24,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,352 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,352 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,352 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:24,354 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af2690e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:24,354 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,355 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:24,355 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,355 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:24,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:24,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:24,358 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:24,358 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:24,358 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:24,358 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:24,358 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:24,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:24,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:24,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:24,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:24,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:24,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:24,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:24,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:24,367 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:24,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:24,368 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/263d17bdd3524ca180c8f7fb7c69ef22 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/263d17bdd3524ca180c8f7fb7c69ef22 2023-04-16 20:54:24,374 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/B of fb9117f6cdea5099d0a91a955c183b16 into 263d17bdd3524ca180c8f7fb7c69ef22(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:24,374 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:24,374 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/B, priority=13, startTime=1681678463921; duration=0sec 2023-04-16 20:54:24,375 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:24,375 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:24,376 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:24,376 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:24,376 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:24,376 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/C is initiating minor compaction (all files) 2023-04-16 20:54:24,376 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/C in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:24,376 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ef98a3dd3ea845e7bce75d140098d5cb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/55b838c1e0ec4b348e2e3b73d675ae94, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/29d6e1861d4f403da60f4cfffa6f0a32] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=36.3 K 2023-04-16 20:54:24,377 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ef98a3dd3ea845e7bce75d140098d5cb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1681678461937 2023-04-16 20:54:24,377 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 55b838c1e0ec4b348e2e3b73d675ae94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681678462362 2023-04-16 20:54:24,377 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 29d6e1861d4f403da60f4cfffa6f0a32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681678463526 2023-04-16 20:54:24,389 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#C#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:24,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/dbd1d91e716e44b4872e4614e6a5a510 2023-04-16 20:54:24,425 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/0b56e9bd6bd14e25b2bb02e7cfe1763f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0b56e9bd6bd14e25b2bb02e7cfe1763f 2023-04-16 20:54:24,429 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/68363b51d36844b4bd1811b193829152 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/68363b51d36844b4bd1811b193829152 2023-04-16 20:54:24,431 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/C of fb9117f6cdea5099d0a91a955c183b16 into 0b56e9bd6bd14e25b2bb02e7cfe1763f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:24,431 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:24,431 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/C, priority=13, startTime=1681678463921; duration=0sec 2023-04-16 20:54:24,431 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:24,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/68363b51d36844b4bd1811b193829152, entries=200, sequenceid=343, filesize=32.3 K 2023-04-16 20:54:24,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/8af48a7e7eb449bda7dd5ecaca4b6592 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8af48a7e7eb449bda7dd5ecaca4b6592 2023-04-16 20:54:24,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8af48a7e7eb449bda7dd5ecaca4b6592, entries=150, sequenceid=343, filesize=11.9 K 2023-04-16 20:54:24,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/dbd1d91e716e44b4872e4614e6a5a510 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/dbd1d91e716e44b4872e4614e6a5a510 2023-04-16 20:54:24,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/dbd1d91e716e44b4872e4614e6a5a510, entries=150, sequenceid=343, filesize=11.9 K 2023-04-16 20:54:24,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fb9117f6cdea5099d0a91a955c183b16 in 496ms, sequenceid=343, compaction requested=false 2023-04-16 20:54:24,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:24,476 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:24,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:24,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:24,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:24,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:24,479 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:24,479 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:24,479 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:24,479 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:24,480 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,480 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,481 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:24,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:24,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:24,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:24,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:24,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:24,483 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:24,483 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:24,483 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:24,483 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 20:54:24,483 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:24,483 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:24,483 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:24,483 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:24,483 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:24,483 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:24,498 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416eaf5814916634d5aba5d22ff92347659 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416eaf5814916634d5aba5d22ff92347659 2023-04-16 20:54:24,503 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=365, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/b0f9cadbfd1e4febb82fcbc4303808d8 2023-04-16 20:54:24,516 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/89a32b167f2e4392ba207ed10525917b 2023-04-16 20:54:24,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:24,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:24,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678524603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678524604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678524706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678524708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,798 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/e006136a852244a2a9ad4dda14d92a63 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e006136a852244a2a9ad4dda14d92a63 2023-04-16 20:54:24,802 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/A of fb9117f6cdea5099d0a91a955c183b16 into e006136a852244a2a9ad4dda14d92a63(size=26.1 K), total size for store is 58.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:24,802 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:24,802 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/A, priority=13, startTime=1681678463921; duration=0sec 2023-04-16 20:54:24,802 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:24,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678524909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:24,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678524911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,926 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/08aaece98f3a469a9f14fa9841d50cfa 2023-04-16 20:54:24,930 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/b0f9cadbfd1e4febb82fcbc4303808d8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/b0f9cadbfd1e4febb82fcbc4303808d8 2023-04-16 20:54:24,934 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/b0f9cadbfd1e4febb82fcbc4303808d8, entries=150, sequenceid=365, filesize=25.4 K 2023-04-16 20:54:24,935 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/89a32b167f2e4392ba207ed10525917b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/89a32b167f2e4392ba207ed10525917b 2023-04-16 20:54:24,938 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/89a32b167f2e4392ba207ed10525917b, entries=150, sequenceid=365, filesize=11.9 K 2023-04-16 20:54:24,939 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/08aaece98f3a469a9f14fa9841d50cfa as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08aaece98f3a469a9f14fa9841d50cfa 2023-04-16 20:54:24,946 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08aaece98f3a469a9f14fa9841d50cfa, entries=150, sequenceid=365, filesize=11.9 K 2023-04-16 20:54:24,947 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fb9117f6cdea5099d0a91a955c183b16 in 464ms, sequenceid=365, compaction requested=true 2023-04-16 20:54:24,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:24,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:24,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:24,950 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,950 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:24,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4868f7ae[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:24,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:24,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,953 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:24,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:24,956 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:24,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:24,962 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:24,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:24,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:24,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:24,962 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:24,965 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,965 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:24,965 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:24,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:24,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:24,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:24,971 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:24,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,971 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:24,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:24,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:24,971 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:24,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:24,971 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:24,971 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:24,971 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:24,971 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,971 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:24,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:24,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:24,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:25,071 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:25,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:25,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:25,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:25,076 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:25,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:25,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:25,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:25,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:25,078 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,078 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:25,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:25,078 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,078 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:25,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:25,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:25,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:25,079 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:25,079 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:25,080 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:25,095 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304160a694d47069444ba8a757068a111bf0e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160a694d47069444ba8a757068a111bf0e 2023-04-16 20:54:25,101 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=378, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/43d0212cf91645b1b47d5fcdfea0dd2d 2023-04-16 20:54:25,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:25,213 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:25,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678525236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678525237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678525341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678525341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,514 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/2e2d6554d66d4c65861463d3adb9e20b 2023-04-16 20:54:25,537 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/08a3d7fedebc4ceaaecf49b602e09b2c 2023-04-16 20:54:25,541 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/43d0212cf91645b1b47d5fcdfea0dd2d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/43d0212cf91645b1b47d5fcdfea0dd2d 2023-04-16 20:54:25,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678525542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678525543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,545 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/43d0212cf91645b1b47d5fcdfea0dd2d, entries=150, sequenceid=378, filesize=25.4 K 2023-04-16 20:54:25,546 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/2e2d6554d66d4c65861463d3adb9e20b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2e2d6554d66d4c65861463d3adb9e20b 2023-04-16 20:54:25,550 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2e2d6554d66d4c65861463d3adb9e20b, entries=150, sequenceid=378, filesize=11.9 K 2023-04-16 20:54:25,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/08a3d7fedebc4ceaaecf49b602e09b2c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08a3d7fedebc4ceaaecf49b602e09b2c 2023-04-16 20:54:25,556 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08a3d7fedebc4ceaaecf49b602e09b2c, entries=150, sequenceid=378, filesize=11.9 K 2023-04-16 20:54:25,557 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fb9117f6cdea5099d0a91a955c183b16 in 477ms, sequenceid=378, compaction requested=true 2023-04-16 20:54:25,557 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:25,557 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:25,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:25,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:25,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:25,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:25,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:25,560 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:25,562 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2df52aea[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:25,562 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:25,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:25,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:25,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:25,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,566 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:25,566 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:25,566 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:25,569 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:25,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:25,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:25,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:25,569 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:25,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:25,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:25,572 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,573 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,577 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,577 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:25,577 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:25,577 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,577 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:25,577 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:25,577 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:25,577 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:25,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:25,577 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:25,577 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:25,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:25,577 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:25,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:25,578 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:25,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:25,678 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:25,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:25,681 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:25,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:25,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:25,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:25,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:25,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:25,683 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:25,683 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,683 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:25,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:25,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:25,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:25,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:25,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:25,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:25,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:25,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:25,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:25,684 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:25,684 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:25,684 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 20:54:25,684 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:25,685 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:25,685 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:25,685 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:25,685 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:25,685 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:25,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:25,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:25,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678525881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678525881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678525985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:25,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:25,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678525985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:26,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59922 deadline: 1681678526075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,082 DEBUG [Thread-994] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:26,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:26,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59886 deadline: 1681678526078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,086 DEBUG [Thread-992] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:26,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:26,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59884 deadline: 1681678526088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,093 DEBUG [Thread-996] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:26,102 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304164b107caef1c14f76907165d1b90c4e8b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164b107caef1c14f76907165d1b90c4e8b 2023-04-16 20:54:26,108 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=398, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/5dfa7d4dc7524ee08b66223a0e0840b4 2023-04-16 20:54:26,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:26,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678526188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:26,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678526188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:26,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678526490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:26,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678526491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,519 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/9ecbc665864646b3a8c58d2f05eac402 2023-04-16 20:54:26,938 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/0d7e4b7ff4a9421395282bf64171486a 2023-04-16 20:54:26,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/5dfa7d4dc7524ee08b66223a0e0840b4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/5dfa7d4dc7524ee08b66223a0e0840b4 2023-04-16 20:54:26,952 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/5dfa7d4dc7524ee08b66223a0e0840b4, entries=150, sequenceid=398, filesize=25.4 K 2023-04-16 20:54:26,953 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/9ecbc665864646b3a8c58d2f05eac402 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/9ecbc665864646b3a8c58d2f05eac402 2023-04-16 20:54:26,959 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/9ecbc665864646b3a8c58d2f05eac402, entries=150, sequenceid=398, filesize=11.9 K 2023-04-16 20:54:26,960 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/0d7e4b7ff4a9421395282bf64171486a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d7e4b7ff4a9421395282bf64171486a 2023-04-16 20:54:26,966 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d7e4b7ff4a9421395282bf64171486a, entries=150, sequenceid=398, filesize=11.9 K 2023-04-16 20:54:26,966 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for fb9117f6cdea5099d0a91a955c183b16 in 1282ms, sequenceid=398, compaction requested=true 2023-04-16 20:54:26,966 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:26,966 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:26,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:26,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:26,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:26,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:26,968 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:26,969 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:26,969 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:26,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:26,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121715d9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:26,971 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:26,971 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:26,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:26,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:26,980 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:26,980 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:26,980 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:26,980 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,980 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:26,982 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:26,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:26,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:26,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:26,984 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:26,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:26,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:26,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:26,984 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:26,986 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:26,986 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:26,986 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:26,987 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:26,988 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,991 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:26,991 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:26,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:26,991 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:26,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:26,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:26,991 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:26,992 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:26,992 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:26,992 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:26,992 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:26,992 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:26,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:26,992 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:26,992 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:26,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:26,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:26,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 20:54:26,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:26,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:26,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:26,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:26,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:26,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:27,037 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041677f5e70b702b4cdd88815d85e40e4f1f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041677f5e70b702b4cdd88815d85e40e4f1f 2023-04-16 20:54:27,046 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=412, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/30cf5028f7e24613b75da150a00a7970 2023-04-16 20:54:27,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:27,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678527042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:27,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678527047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,092 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:27,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:27,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:27,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:27,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:27,097 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:27,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:27,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:27,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,098 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:27,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:27,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:27,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:27,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:27,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:27,100 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,101 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:27,101 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:27,101 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:27,103 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b74b48c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:27,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:27,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,106 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,106 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:27,106 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:27,106 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:27,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:27,107 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:27,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:27,110 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:27,110 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:27,110 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:27,110 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:27,110 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:27,111 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,111 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,111 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,112 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,113 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:27,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:27,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:27,116 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,116 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:27,117 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:27,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,117 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:27,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678527148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:27,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678527149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,217 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:27,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:27,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:27,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:27,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:27,220 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:27,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:27,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:27,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,221 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,222 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,222 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,222 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,222 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:27,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:27,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:27,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:27,224 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,224 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:27,224 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:27,224 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:27,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,225 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,225 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,225 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:27,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442effcb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:27,227 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:27,227 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,228 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:27,228 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,228 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:27,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,230 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:27,230 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,230 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:27,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:27,233 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:27,234 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:27,234 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:27,234 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:27,234 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:27,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,235 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,237 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,238 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,238 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,238 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:27,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:27,238 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,238 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,238 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,239 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:27,239 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:27,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,339 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:27,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:27,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:27,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:27,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:27,343 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:27,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:27,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:27,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,344 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:27,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:27,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:27,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:27,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:27,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:27,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:27,347 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,347 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:27,347 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:27,347 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:27,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:27,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:27,347 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:27,349 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,349 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,350 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,350 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:27,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40882fde[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:27,351 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:27,351 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,352 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:27,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:27,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:27,353 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,353 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:27,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:27,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678527352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:27,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678527353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:27,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:27,359 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:27,359 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:27,359 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:27,359 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:27,359 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:27,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,361 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,370 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:27,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:27,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,371 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:27,371 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:27,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:27,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/dccf472506ae4a12aae73ba934d92006 2023-04-16 20:54:27,471 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:27,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:27,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:27,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:27,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:27,474 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:27,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:27,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:27,475 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,475 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,476 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:27,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:27,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:27,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:27,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:27,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:27,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:27,477 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,477 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:27,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:27,478 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:27,478 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:27,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:27,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:27,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:27,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:27,482 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:27,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a9b43cb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:27,482 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,484 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,484 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:27,485 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:27,485 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:27,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:27,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,486 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:27,486 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,486 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:27,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:27,488 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:27,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:27,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:27,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:27,488 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:27,489 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,490 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,490 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,491 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,492 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:27,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:27,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,494 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:27,494 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:27,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,594 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:27,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:27,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:27,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:27,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:27,598 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:27,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:27,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:27,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,600 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,601 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,601 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:27,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:27,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:27,602 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,602 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:27,602 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:27,602 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:27,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,604 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:27,605 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:27,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4255ac0[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:27,605 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,606 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,607 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:27,607 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:27,607 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:27,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:27,608 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:27,608 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,608 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,608 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:27,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:27,610 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:27,610 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:27,610 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:27,610 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:27,610 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:27,613 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,613 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,613 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,613 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,618 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:27,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:27,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,618 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:27,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,619 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:27,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:27,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678527654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:27,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678527655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,719 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:27,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:27,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:27,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:27,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:27,722 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:27,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:27,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:27,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,724 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,724 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,724 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:27,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:27,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:27,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:27,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:27,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:27,725 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,725 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:27,725 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:27,726 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:27,727 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:27,729 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:27,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c63d28[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:27,729 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,731 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,731 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:27,731 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:27,731 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:27,731 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,731 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:27,732 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:27,732 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,732 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,732 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:27,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:27,734 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:27,735 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:27,735 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:27,735 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:27,735 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:27,736 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,738 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,741 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:27,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:27,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,748 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:27,748 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:27,848 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:27,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:27,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:27,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:27,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:27,851 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:27,851 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,851 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:27,851 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:27,852 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,852 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,853 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,853 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,853 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:27,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:27,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:27,854 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:27,854 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,854 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:27,854 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:27,854 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:27,856 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,856 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:27,856 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:27,856 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,856 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:27,856 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:27,856 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,856 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:27,856 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:27,857 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@363102d1[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:27,859 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:27,859 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,860 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:27,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:27,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:27,860 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,861 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:27,863 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,863 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,863 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,863 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:27,865 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:27,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:27,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:27,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:27,865 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:27,867 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,867 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:27,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,867 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:27,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,867 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,868 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:27,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:27,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:27,870 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:27,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:27,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,871 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:27,871 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:27,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:27,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/5b9dbba19bf749ca824636b7aaf260da 2023-04-16 20:54:27,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/30cf5028f7e24613b75da150a00a7970 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/30cf5028f7e24613b75da150a00a7970 2023-04-16 20:54:27,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/30cf5028f7e24613b75da150a00a7970, entries=200, sequenceid=412, filesize=32.3 K 2023-04-16 20:54:27,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/dccf472506ae4a12aae73ba934d92006 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/dccf472506ae4a12aae73ba934d92006 2023-04-16 20:54:27,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/dccf472506ae4a12aae73ba934d92006, entries=150, sequenceid=412, filesize=11.9 K 2023-04-16 20:54:27,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/5b9dbba19bf749ca824636b7aaf260da as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/5b9dbba19bf749ca824636b7aaf260da 2023-04-16 20:54:27,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/5b9dbba19bf749ca824636b7aaf260da, entries=150, sequenceid=412, filesize=11.9 K 2023-04-16 20:54:27,892 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for fb9117f6cdea5099d0a91a955c183b16 in 895ms, sequenceid=412, compaction requested=true 2023-04-16 20:54:27,892 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:27,893 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:27,893 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:27,893 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:54:27,893 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:54:27,893 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:54:27,895 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 171038 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:54:27,895 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/A is initiating minor compaction (all files) 2023-04-16 20:54:27,895 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/A in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,895 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e006136a852244a2a9ad4dda14d92a63, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/68363b51d36844b4bd1811b193829152, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/b0f9cadbfd1e4febb82fcbc4303808d8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/43d0212cf91645b1b47d5fcdfea0dd2d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/5dfa7d4dc7524ee08b66223a0e0840b4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/30cf5028f7e24613b75da150a00a7970] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=167.0 K 2023-04-16 20:54:27,896 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:54:27,896 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/B is initiating minor compaction (all files) 2023-04-16 20:54:27,896 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/B in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,896 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/263d17bdd3524ca180c8f7fb7c69ef22, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8af48a7e7eb449bda7dd5ecaca4b6592, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/89a32b167f2e4392ba207ed10525917b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2e2d6554d66d4c65861463d3adb9e20b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/9ecbc665864646b3a8c58d2f05eac402, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/dccf472506ae4a12aae73ba934d92006] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=72.1 K 2023-04-16 20:54:27,896 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e006136a852244a2a9ad4dda14d92a63, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681678463526 2023-04-16 20:54:27,897 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 68363b51d36844b4bd1811b193829152, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681678463848 2023-04-16 20:54:27,897 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 263d17bdd3524ca180c8f7fb7c69ef22, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681678463526 2023-04-16 20:54:27,897 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b0f9cadbfd1e4febb82fcbc4303808d8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681678463977 2023-04-16 20:54:27,897 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8af48a7e7eb449bda7dd5ecaca4b6592, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681678463848 2023-04-16 20:54:27,898 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 89a32b167f2e4392ba207ed10525917b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681678463977 2023-04-16 20:54:27,898 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 43d0212cf91645b1b47d5fcdfea0dd2d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681678464597 2023-04-16 20:54:27,898 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2e2d6554d66d4c65861463d3adb9e20b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681678464597 2023-04-16 20:54:27,898 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5dfa7d4dc7524ee08b66223a0e0840b4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681678465225 2023-04-16 20:54:27,898 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9ecbc665864646b3a8c58d2f05eac402, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681678465225 2023-04-16 20:54:27,899 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 30cf5028f7e24613b75da150a00a7970, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1681678465854 2023-04-16 20:54:27,899 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting dccf472506ae4a12aae73ba934d92006, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1681678465854 2023-04-16 20:54:27,912 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#B#compaction#214 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:27,926 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#A#compaction#213 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:27,946 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/e7e86c8e1b0a4cf3a2f199d23c884e47 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e7e86c8e1b0a4cf3a2f199d23c884e47 2023-04-16 20:54:27,951 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/A of fb9117f6cdea5099d0a91a955c183b16 into e7e86c8e1b0a4cf3a2f199d23c884e47(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:27,951 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:27,951 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/A, priority=10, startTime=1681678467893; duration=0sec 2023-04-16 20:54:27,951 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:27,951 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:54:27,953 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:54:27,953 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): fb9117f6cdea5099d0a91a955c183b16/C is initiating minor compaction (all files) 2023-04-16 20:54:27,953 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb9117f6cdea5099d0a91a955c183b16/C in TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,953 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0b56e9bd6bd14e25b2bb02e7cfe1763f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/dbd1d91e716e44b4872e4614e6a5a510, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08aaece98f3a469a9f14fa9841d50cfa, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08a3d7fedebc4ceaaecf49b602e09b2c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d7e4b7ff4a9421395282bf64171486a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/5b9dbba19bf749ca824636b7aaf260da] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp, totalSize=72.1 K 2023-04-16 20:54:27,953 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0b56e9bd6bd14e25b2bb02e7cfe1763f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681678463526 2023-04-16 20:54:27,955 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting dbd1d91e716e44b4872e4614e6a5a510, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681678463848 2023-04-16 20:54:27,955 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 08aaece98f3a469a9f14fa9841d50cfa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681678463977 2023-04-16 20:54:27,955 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 08a3d7fedebc4ceaaecf49b602e09b2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681678464597 2023-04-16 20:54:27,956 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0d7e4b7ff4a9421395282bf64171486a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681678465225 2023-04-16 20:54:27,956 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5b9dbba19bf749ca824636b7aaf260da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1681678465854 2023-04-16 20:54:27,970 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): fb9117f6cdea5099d0a91a955c183b16#C#compaction#215 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:27,971 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:27,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:27,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:27,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:27,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:27,980 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:27,980 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,980 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:27,980 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:27,980 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,981 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,982 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:27,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:27,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:27,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:27,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:27,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:27,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:27,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:27,992 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:27,992 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:27,992 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:27,992 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:27,992 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:27,992 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:27,992 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:27,993 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-16 20:54:27,993 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:27,993 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:27,993 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:27,993 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:27,993 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:27,993 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:28,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:28,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:28,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:28,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678528176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:28,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678528178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:28,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678528280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:28,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678528282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,333 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/33d6a82cb5b947608d4905bfa3eba036 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/33d6a82cb5b947608d4905bfa3eba036 2023-04-16 20:54:28,337 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/B of fb9117f6cdea5099d0a91a955c183b16 into 33d6a82cb5b947608d4905bfa3eba036(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:28,337 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:28,337 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/B, priority=10, startTime=1681678467893; duration=0sec 2023-04-16 20:54:28,337 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:28,394 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/72d85d5ecc0047bd83da6db25c373a96 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/72d85d5ecc0047bd83da6db25c373a96 2023-04-16 20:54:28,398 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fb9117f6cdea5099d0a91a955c183b16/C of fb9117f6cdea5099d0a91a955c183b16 into 72d85d5ecc0047bd83da6db25c373a96(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:28,398 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:28,398 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16., storeName=fb9117f6cdea5099d0a91a955c183b16/C, priority=10, startTime=1681678467893; duration=0sec 2023-04-16 20:54:28,398 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:28,415 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416f6935be04eda41048ca49ce4a8c495a7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f6935be04eda41048ca49ce4a8c495a7 2023-04-16 20:54:28,419 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=431, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/c36f6711ca0645d7bb34f9886b6591a9 2023-04-16 20:54:28,432 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/04ad39808b3f4c988d202c7623d7adc5 2023-04-16 20:54:28,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:28,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678528482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:28,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678528487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:28,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678528785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678528790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,853 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/fa60d522092a4380a2010cb4e4445384 2023-04-16 20:54:28,858 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/c36f6711ca0645d7bb34f9886b6591a9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/c36f6711ca0645d7bb34f9886b6591a9 2023-04-16 20:54:28,862 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/c36f6711ca0645d7bb34f9886b6591a9, entries=150, sequenceid=431, filesize=25.4 K 2023-04-16 20:54:28,863 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/04ad39808b3f4c988d202c7623d7adc5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/04ad39808b3f4c988d202c7623d7adc5 2023-04-16 20:54:28,867 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/04ad39808b3f4c988d202c7623d7adc5, entries=150, sequenceid=431, filesize=11.9 K 2023-04-16 20:54:28,868 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/fa60d522092a4380a2010cb4e4445384 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fa60d522092a4380a2010cb4e4445384 2023-04-16 20:54:28,878 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fa60d522092a4380a2010cb4e4445384, entries=150, sequenceid=431, filesize=11.9 K 2023-04-16 20:54:28,879 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for fb9117f6cdea5099d0a91a955c183b16 in 886ms, sequenceid=431, compaction requested=false 2023-04-16 20:54:28,879 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:28,879 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:28,882 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6848ac7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:28,884 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:28,884 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:28,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:28,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:28,885 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:28,885 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:28,886 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:28,886 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,886 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:28,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:28,900 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:28,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:28,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:28,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:28,900 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:28,902 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:28,902 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:28,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:28,902 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:28,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:28,903 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:28,904 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:28,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:28,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:28,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:28,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:28,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:28,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:28,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:28,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:28,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:28,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:28,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:28,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:28,908 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:28,909 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:28,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:28,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:29,009 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:29,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:29,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:29,012 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:29,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:29,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:29,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:29,013 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:29,013 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:29,014 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:29,014 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:29,015 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,015 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:29,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:29,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:29,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:29,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:29,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:29,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:29,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. started... 2023-04-16 20:54:29,017 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 20:54:29,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:29,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:29,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:29,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:29,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:29,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:29,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:29,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. as already flushing 2023-04-16 20:54:29,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:29,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678529335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:29,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678529336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,432 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416261465b8fa024b348b69ac26b38a4135 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416261465b8fa024b348b69ac26b38a4135 2023-04-16 20:54:29,437 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=447, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/98155403913f45fbb6ae1de04fae2e7a 2023-04-16 20:54:29,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:29,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678529437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:29,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678529439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,460 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/cb4e63f7a4d4457fa699fc6d9ac35820 2023-04-16 20:54:29,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59906 deadline: 1681678529640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59890 deadline: 1681678529641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,877 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/6fe14c038785405b83af6547d39d83c9 2023-04-16 20:54:29,882 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/98155403913f45fbb6ae1de04fae2e7a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/98155403913f45fbb6ae1de04fae2e7a 2023-04-16 20:54:29,885 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/98155403913f45fbb6ae1de04fae2e7a, entries=100, sequenceid=447, filesize=18.6 K 2023-04-16 20:54:29,886 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/cb4e63f7a4d4457fa699fc6d9ac35820 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/cb4e63f7a4d4457fa699fc6d9ac35820 2023-04-16 20:54:29,890 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/cb4e63f7a4d4457fa699fc6d9ac35820, entries=100, sequenceid=447, filesize=9.5 K 2023-04-16 20:54:29,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/6fe14c038785405b83af6547d39d83c9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/6fe14c038785405b83af6547d39d83c9 2023-04-16 20:54:29,894 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/6fe14c038785405b83af6547d39d83c9, entries=100, sequenceid=447, filesize=9.5 K 2023-04-16 20:54:29,895 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for fb9117f6cdea5099d0a91a955c183b16 in 878ms, sequenceid=447, compaction requested=true 2023-04-16 20:54:29,895 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:29,895 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:29,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:29,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:29,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:29,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:29,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28d9338d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:29,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:29,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:29,902 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:29,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:29,902 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,902 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:29,905 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:29,905 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,905 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:29,905 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:29,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:29,907 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:29,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:29,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:29,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:29,908 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:29,909 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:29,909 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:29,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:29,909 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:29,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:29,911 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:29,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,916 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,916 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:29,916 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:29,916 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:29,916 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:29,916 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:29,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:29,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:29,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:29,916 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:29,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:29,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:29,917 DEBUG [Thread-1000] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:29,916 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:29,917 DEBUG [Thread-1000] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:29,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:29,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:29,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:29,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:29,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 20:54:29,947 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:29,947 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:29,947 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:29,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:29,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:29,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:29,954 DEBUG [Thread-998] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:54578 2023-04-16 20:54:29,954 DEBUG [Thread-998] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:29,954 DEBUG [Thread-1007] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:54578 2023-04-16 20:54:29,954 DEBUG [Thread-1007] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:29,955 DEBUG [Thread-1005] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:54578 2023-04-16 20:54:29,955 DEBUG [Thread-1005] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:29,956 DEBUG [Thread-990] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34135972 to 127.0.0.1:54578 2023-04-16 20:54:29,956 DEBUG [Thread-990] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:29,956 DEBUG [Thread-1001] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x498694cb to 127.0.0.1:54578 2023-04-16 20:54:29,956 DEBUG [Thread-1001] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:29,956 DEBUG [Thread-1003] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:54578 2023-04-16 20:54:29,956 DEBUG [Thread-1003] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:30,017 DEBUG [Thread-1000] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:30,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:30,085 DEBUG [Thread-994] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:54578 2023-04-16 20:54:30,085 DEBUG [Thread-994] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:30,112 DEBUG [Thread-996] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x370947cb to 127.0.0.1:54578 2023-04-16 20:54:30,112 DEBUG [Thread-996] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:30,112 DEBUG [Thread-992] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02bab6b2 to 127.0.0.1:54578 2023-04-16 20:54:30,112 DEBUG [Thread-992] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 5300 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 5241 2023-04-16 20:54:30,112 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 20:54:30,113 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 2220 2023-04-16 20:54:30,113 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 6660 rows 2023-04-16 20:54:30,113 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 2224 2023-04-16 20:54:30,113 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 6672 rows 2023-04-16 20:54:30,113 INFO [Listener at localhost/45179] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 20:54:30,113 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52274650 to 127.0.0.1:54578 2023-04-16 20:54:30,113 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:30,115 INFO [Listener at localhost/45179] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 20:54:30,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 20:54:30,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:30,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 20:54:30,120 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678470120"}]},"ts":"1681678470120"} 2023-04-16 20:54:30,121 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 20:54:30,122 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 20:54:30,123 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, UNASSIGN}] 2023-04-16 20:54:30,124 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, UNASSIGN 2023-04-16 20:54:30,125 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fb9117f6cdea5099d0a91a955c183b16, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:30,125 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678470125"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678470125"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678470125"}]},"ts":"1681678470125"} 2023-04-16 20:54:30,125 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:54:30,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 20:54:30,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:30,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fb9117f6cdea5099d0a91a955c183b16, disabling compactions & flushes 2023-04-16 20:54:30,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:30,363 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416810219fd19a74b0d9351f3369c08d83a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416810219fd19a74b0d9351f3369c08d83a 2023-04-16 20:54:30,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 20:54:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 20:54:30,768 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=467, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/dadd9009b7d74af29ab0cde675380276 2023-04-16 20:54:31,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/29e599e196a14634bf3f3a521ee82a81 2023-04-16 20:54:31,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 20:54:31,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/292107dbb3284b39ae91e723699e0e62 2023-04-16 20:54:31,590 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/dadd9009b7d74af29ab0cde675380276 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/dadd9009b7d74af29ab0cde675380276 2023-04-16 20:54:31,593 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/dadd9009b7d74af29ab0cde675380276, entries=200, sequenceid=467, filesize=32.3 K 2023-04-16 20:54:31,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/29e599e196a14634bf3f3a521ee82a81 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/29e599e196a14634bf3f3a521ee82a81 2023-04-16 20:54:31,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/29e599e196a14634bf3f3a521ee82a81, entries=150, sequenceid=467, filesize=11.9 K 2023-04-16 20:54:31,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/292107dbb3284b39ae91e723699e0e62 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/292107dbb3284b39ae91e723699e0e62 2023-04-16 20:54:31,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/292107dbb3284b39ae91e723699e0e62, entries=150, sequenceid=467, filesize=11.9 K 2023-04-16 20:54:31,602 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=53.67 KB/54960 for fb9117f6cdea5099d0a91a955c183b16 in 1655ms, sequenceid=467, compaction requested=true 2023-04-16 20:54:31,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:31,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:31,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:31,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. after waiting 0 ms 2023-04-16 20:54:31,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:31,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:31,602 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. because compaction request was cancelled 2023-04-16 20:54:31,602 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. because compaction request was cancelled 2023-04-16 20:54:31,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:31,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fb9117f6cdea5099d0a91a955c183b16 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:54:31,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:31,602 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. because compaction request was cancelled 2023-04-16 20:54:31,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=A 2023-04-16 20:54:31,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:31,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=B 2023-04-16 20:54:31,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:31,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb9117f6cdea5099d0a91a955c183b16, store=C 2023-04-16 20:54:31,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:32,014 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304167f366193450941ae9385c6bb607407f1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167f366193450941ae9385c6bb607407f1 2023-04-16 20:54:32,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 20:54:32,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=478, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/0ea69b2c16564cce969d73f976416c8d 2023-04-16 20:54:32,827 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/c03769b5afb648a6912cf9b8c479487a 2023-04-16 20:54:33,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/93478278e8584fff8ef62d42379bcdfa 2023-04-16 20:54:33,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/A/0ea69b2c16564cce969d73f976416c8d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/0ea69b2c16564cce969d73f976416c8d 2023-04-16 20:54:33,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/0ea69b2c16564cce969d73f976416c8d, entries=150, sequenceid=478, filesize=25.4 K 2023-04-16 20:54:33,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/B/c03769b5afb648a6912cf9b8c479487a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/c03769b5afb648a6912cf9b8c479487a 2023-04-16 20:54:33,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/c03769b5afb648a6912cf9b8c479487a, entries=150, sequenceid=478, filesize=11.9 K 2023-04-16 20:54:33,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/.tmp/C/93478278e8584fff8ef62d42379bcdfa as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/93478278e8584fff8ef62d42379bcdfa 2023-04-16 20:54:33,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/93478278e8584fff8ef62d42379bcdfa, entries=150, sequenceid=478, filesize=11.9 K 2023-04-16 20:54:33,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for fb9117f6cdea5099d0a91a955c183b16 in 1649ms, sequenceid=478, compaction requested=true 2023-04-16 20:54:33,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3e62900126cd48f5a76ffcd761e851d1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e0fa8fe03954f4ca49e521003b8e5df, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e9fe7c707b704b5fb652fc4668f3ff4c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/9dabf7e4a74c4f989fe22d6ec00950cb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e9734464824446f825a9c438f308d7b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/0f154001ddcd48c18d6b88f759d3d7b2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/8e0eaa006d3f488faffd1979affde6c2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/ecd871861d0640b893dbc7d335d59501, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e55bf89b76ad42a9963bbf3ca55de38b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/2784fb5e5b7d4fff90ff6d5449a5a6c2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1341abcbb3264a258bc140ee7a87f4b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/61318119d3694764b5d5cce87fc5565c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/4de6de21c91a4b1b807f268d87724ae8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/c5391745c14c4758aa5223664492d1e1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3b3a7f497130477fa3a51e32f0da4c39, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1fa4a5acd93c468ca67389849e09275a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/73d93ecd56d340e784f21d582aa6f2e3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/a4936ab52985420aaead9bc00501cb78, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1d8c7eb2ecb048218693dff95bef8280, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/32e53ee2d554405e87784e78585899c3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e006136a852244a2a9ad4dda14d92a63, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/68363b51d36844b4bd1811b193829152, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/b0f9cadbfd1e4febb82fcbc4303808d8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/43d0212cf91645b1b47d5fcdfea0dd2d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/5dfa7d4dc7524ee08b66223a0e0840b4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/30cf5028f7e24613b75da150a00a7970] to archive 2023-04-16 20:54:33,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:54:33,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3e62900126cd48f5a76ffcd761e851d1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3e62900126cd48f5a76ffcd761e851d1 2023-04-16 20:54:33,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e0fa8fe03954f4ca49e521003b8e5df to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e0fa8fe03954f4ca49e521003b8e5df 2023-04-16 20:54:33,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e9fe7c707b704b5fb652fc4668f3ff4c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e9fe7c707b704b5fb652fc4668f3ff4c 2023-04-16 20:54:33,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/9dabf7e4a74c4f989fe22d6ec00950cb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/9dabf7e4a74c4f989fe22d6ec00950cb 2023-04-16 20:54:33,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e9734464824446f825a9c438f308d7b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/6e9734464824446f825a9c438f308d7b 2023-04-16 20:54:33,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/0f154001ddcd48c18d6b88f759d3d7b2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/0f154001ddcd48c18d6b88f759d3d7b2 2023-04-16 20:54:33,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/8e0eaa006d3f488faffd1979affde6c2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/8e0eaa006d3f488faffd1979affde6c2 2023-04-16 20:54:33,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/ecd871861d0640b893dbc7d335d59501 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/ecd871861d0640b893dbc7d335d59501 2023-04-16 20:54:33,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e55bf89b76ad42a9963bbf3ca55de38b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e55bf89b76ad42a9963bbf3ca55de38b 2023-04-16 20:54:33,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/2784fb5e5b7d4fff90ff6d5449a5a6c2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/2784fb5e5b7d4fff90ff6d5449a5a6c2 2023-04-16 20:54:33,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1341abcbb3264a258bc140ee7a87f4b0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1341abcbb3264a258bc140ee7a87f4b0 2023-04-16 20:54:33,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/61318119d3694764b5d5cce87fc5565c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/61318119d3694764b5d5cce87fc5565c 2023-04-16 20:54:33,287 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/4de6de21c91a4b1b807f268d87724ae8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/4de6de21c91a4b1b807f268d87724ae8 2023-04-16 20:54:33,288 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/c5391745c14c4758aa5223664492d1e1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/c5391745c14c4758aa5223664492d1e1 2023-04-16 20:54:33,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3b3a7f497130477fa3a51e32f0da4c39 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/3b3a7f497130477fa3a51e32f0da4c39 2023-04-16 20:54:33,290 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1fa4a5acd93c468ca67389849e09275a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1fa4a5acd93c468ca67389849e09275a 2023-04-16 20:54:33,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/73d93ecd56d340e784f21d582aa6f2e3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/73d93ecd56d340e784f21d582aa6f2e3 2023-04-16 20:54:33,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/a4936ab52985420aaead9bc00501cb78 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/a4936ab52985420aaead9bc00501cb78 2023-04-16 20:54:33,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1d8c7eb2ecb048218693dff95bef8280 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/1d8c7eb2ecb048218693dff95bef8280 2023-04-16 20:54:33,295 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/32e53ee2d554405e87784e78585899c3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/32e53ee2d554405e87784e78585899c3 2023-04-16 20:54:33,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e006136a852244a2a9ad4dda14d92a63 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e006136a852244a2a9ad4dda14d92a63 2023-04-16 20:54:33,297 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/68363b51d36844b4bd1811b193829152 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/68363b51d36844b4bd1811b193829152 2023-04-16 20:54:33,298 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/b0f9cadbfd1e4febb82fcbc4303808d8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/b0f9cadbfd1e4febb82fcbc4303808d8 2023-04-16 20:54:33,299 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/43d0212cf91645b1b47d5fcdfea0dd2d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/43d0212cf91645b1b47d5fcdfea0dd2d 2023-04-16 20:54:33,300 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/5dfa7d4dc7524ee08b66223a0e0840b4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/5dfa7d4dc7524ee08b66223a0e0840b4 2023-04-16 20:54:33,301 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/30cf5028f7e24613b75da150a00a7970 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/30cf5028f7e24613b75da150a00a7970 2023-04-16 20:54:33,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/ca264b0ca7f44ea3856041cb8c1587cc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/da5191fd279144ecad1f1c22c59f4fd5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/20bd66316f5f47398652c75f1764c516, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/d30e5455f4604e08bad54e614b9dd957, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2424eb54132341c69419f1246b764e21, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/24ebcb820b014430aa54d892f7c13fa5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7530be2bd5df49d7a7a82406eca96582, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/96f27d70b7e44c62b13c82976e1e971f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/947f492c21bc4fca89e2bef0bea41c98, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/3fba0951dd3441afb3b7dc379c81464d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/6579983e40a24f46abd042f1f38f3842, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/4f40b57b0e9f49ec9e4d0a4306411de1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8e2999c87bf344439c1224d208350621, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/67a9ddaf5fd84d25bde3ae342d5c95fc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e4c6057e9e44490b976ba91ec15a6766, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/b27fb05c301443b582627be3e2e9dd68, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e9c21d6ab96c41f09a7dac68110b1611, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7a411c377eed478a935bbd9a7e3e2d40, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/097d8ef7a80d45079a98cf4871a69376, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/263d17bdd3524ca180c8f7fb7c69ef22, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8250cb43a89d42bf91d03dd098427a79, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8af48a7e7eb449bda7dd5ecaca4b6592, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/89a32b167f2e4392ba207ed10525917b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2e2d6554d66d4c65861463d3adb9e20b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/9ecbc665864646b3a8c58d2f05eac402, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/dccf472506ae4a12aae73ba934d92006] to archive 2023-04-16 20:54:33,304 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:54:33,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/ca264b0ca7f44ea3856041cb8c1587cc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/ca264b0ca7f44ea3856041cb8c1587cc 2023-04-16 20:54:33,306 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/da5191fd279144ecad1f1c22c59f4fd5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/da5191fd279144ecad1f1c22c59f4fd5 2023-04-16 20:54:33,308 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/20bd66316f5f47398652c75f1764c516 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/20bd66316f5f47398652c75f1764c516 2023-04-16 20:54:33,309 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/d30e5455f4604e08bad54e614b9dd957 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/d30e5455f4604e08bad54e614b9dd957 2023-04-16 20:54:33,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2424eb54132341c69419f1246b764e21 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2424eb54132341c69419f1246b764e21 2023-04-16 20:54:33,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/24ebcb820b014430aa54d892f7c13fa5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/24ebcb820b014430aa54d892f7c13fa5 2023-04-16 20:54:33,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7530be2bd5df49d7a7a82406eca96582 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7530be2bd5df49d7a7a82406eca96582 2023-04-16 20:54:33,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/96f27d70b7e44c62b13c82976e1e971f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/96f27d70b7e44c62b13c82976e1e971f 2023-04-16 20:54:33,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/947f492c21bc4fca89e2bef0bea41c98 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/947f492c21bc4fca89e2bef0bea41c98 2023-04-16 20:54:33,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/3fba0951dd3441afb3b7dc379c81464d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/3fba0951dd3441afb3b7dc379c81464d 2023-04-16 20:54:33,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/6579983e40a24f46abd042f1f38f3842 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/6579983e40a24f46abd042f1f38f3842 2023-04-16 20:54:33,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/4f40b57b0e9f49ec9e4d0a4306411de1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/4f40b57b0e9f49ec9e4d0a4306411de1 2023-04-16 20:54:33,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8e2999c87bf344439c1224d208350621 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8e2999c87bf344439c1224d208350621 2023-04-16 20:54:33,319 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/67a9ddaf5fd84d25bde3ae342d5c95fc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/67a9ddaf5fd84d25bde3ae342d5c95fc 2023-04-16 20:54:33,320 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e4c6057e9e44490b976ba91ec15a6766 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e4c6057e9e44490b976ba91ec15a6766 2023-04-16 20:54:33,321 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/b27fb05c301443b582627be3e2e9dd68 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/b27fb05c301443b582627be3e2e9dd68 2023-04-16 20:54:33,322 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e9c21d6ab96c41f09a7dac68110b1611 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/e9c21d6ab96c41f09a7dac68110b1611 2023-04-16 20:54:33,323 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7a411c377eed478a935bbd9a7e3e2d40 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/7a411c377eed478a935bbd9a7e3e2d40 2023-04-16 20:54:33,324 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/097d8ef7a80d45079a98cf4871a69376 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/097d8ef7a80d45079a98cf4871a69376 2023-04-16 20:54:33,325 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/263d17bdd3524ca180c8f7fb7c69ef22 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/263d17bdd3524ca180c8f7fb7c69ef22 2023-04-16 20:54:33,325 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8250cb43a89d42bf91d03dd098427a79 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8250cb43a89d42bf91d03dd098427a79 2023-04-16 20:54:33,326 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8af48a7e7eb449bda7dd5ecaca4b6592 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/8af48a7e7eb449bda7dd5ecaca4b6592 2023-04-16 20:54:33,328 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/89a32b167f2e4392ba207ed10525917b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/89a32b167f2e4392ba207ed10525917b 2023-04-16 20:54:33,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2e2d6554d66d4c65861463d3adb9e20b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/2e2d6554d66d4c65861463d3adb9e20b 2023-04-16 20:54:33,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/9ecbc665864646b3a8c58d2f05eac402 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/9ecbc665864646b3a8c58d2f05eac402 2023-04-16 20:54:33,331 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/dccf472506ae4a12aae73ba934d92006 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/dccf472506ae4a12aae73ba934d92006 2023-04-16 20:54:33,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d7c3dcc9a346491cb1b3d4cef1ea6cfb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/75847f249e1848c1be8ebcb6e718e826, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/38158b5b40564aedbd4813460054f4eb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3fbda78375b84df69958a30ab8d37b9b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3829eeacda8d4029ab0f352426d0e96a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/394dc016a64a4111bd52439c629f8d66, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/19975585ce144395abd8cbc5a110f39d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/90bc490b4a284f0e90b2bab0d7a1e754, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d35ddebce34488bb06e1ca2d8456204, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d9ce872fb1d44485a8c594606c319e5e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/51893fdaca184000b8d6a86b30153a8a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/e4e44be0efbf4350b8a541d11bd21360, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/b469c4950a1445d79fb1a12986913b3d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fff9cab247c1400691f2484b17f1eb55, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/306d888feb7c4725bca04e1e389842bc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/02a2e3fd1f604515a99d9408455e32f4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ef98a3dd3ea845e7bce75d140098d5cb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ca8bc94269994bf7a0153383df627143, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/55b838c1e0ec4b348e2e3b73d675ae94, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0b56e9bd6bd14e25b2bb02e7cfe1763f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/29d6e1861d4f403da60f4cfffa6f0a32, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/dbd1d91e716e44b4872e4614e6a5a510, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08aaece98f3a469a9f14fa9841d50cfa, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08a3d7fedebc4ceaaecf49b602e09b2c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d7e4b7ff4a9421395282bf64171486a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/5b9dbba19bf749ca824636b7aaf260da] to archive 2023-04-16 20:54:33,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:54:33,335 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d7c3dcc9a346491cb1b3d4cef1ea6cfb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d7c3dcc9a346491cb1b3d4cef1ea6cfb 2023-04-16 20:54:33,336 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/75847f249e1848c1be8ebcb6e718e826 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/75847f249e1848c1be8ebcb6e718e826 2023-04-16 20:54:33,337 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/38158b5b40564aedbd4813460054f4eb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/38158b5b40564aedbd4813460054f4eb 2023-04-16 20:54:33,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3fbda78375b84df69958a30ab8d37b9b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3fbda78375b84df69958a30ab8d37b9b 2023-04-16 20:54:33,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3829eeacda8d4029ab0f352426d0e96a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/3829eeacda8d4029ab0f352426d0e96a 2023-04-16 20:54:33,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/394dc016a64a4111bd52439c629f8d66 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/394dc016a64a4111bd52439c629f8d66 2023-04-16 20:54:33,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/19975585ce144395abd8cbc5a110f39d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/19975585ce144395abd8cbc5a110f39d 2023-04-16 20:54:33,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/90bc490b4a284f0e90b2bab0d7a1e754 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/90bc490b4a284f0e90b2bab0d7a1e754 2023-04-16 20:54:33,343 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d35ddebce34488bb06e1ca2d8456204 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d35ddebce34488bb06e1ca2d8456204 2023-04-16 20:54:33,344 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d9ce872fb1d44485a8c594606c319e5e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/d9ce872fb1d44485a8c594606c319e5e 2023-04-16 20:54:33,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/51893fdaca184000b8d6a86b30153a8a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/51893fdaca184000b8d6a86b30153a8a 2023-04-16 20:54:33,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/e4e44be0efbf4350b8a541d11bd21360 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/e4e44be0efbf4350b8a541d11bd21360 2023-04-16 20:54:33,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/b469c4950a1445d79fb1a12986913b3d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/b469c4950a1445d79fb1a12986913b3d 2023-04-16 20:54:33,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fff9cab247c1400691f2484b17f1eb55 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fff9cab247c1400691f2484b17f1eb55 2023-04-16 20:54:33,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/306d888feb7c4725bca04e1e389842bc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/306d888feb7c4725bca04e1e389842bc 2023-04-16 20:54:33,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/02a2e3fd1f604515a99d9408455e32f4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/02a2e3fd1f604515a99d9408455e32f4 2023-04-16 20:54:33,350 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ef98a3dd3ea845e7bce75d140098d5cb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ef98a3dd3ea845e7bce75d140098d5cb 2023-04-16 20:54:33,351 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ca8bc94269994bf7a0153383df627143 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/ca8bc94269994bf7a0153383df627143 2023-04-16 20:54:33,352 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/55b838c1e0ec4b348e2e3b73d675ae94 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/55b838c1e0ec4b348e2e3b73d675ae94 2023-04-16 20:54:33,353 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0b56e9bd6bd14e25b2bb02e7cfe1763f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0b56e9bd6bd14e25b2bb02e7cfe1763f 2023-04-16 20:54:33,354 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/29d6e1861d4f403da60f4cfffa6f0a32 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/29d6e1861d4f403da60f4cfffa6f0a32 2023-04-16 20:54:33,355 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/dbd1d91e716e44b4872e4614e6a5a510 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/dbd1d91e716e44b4872e4614e6a5a510 2023-04-16 20:54:33,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08aaece98f3a469a9f14fa9841d50cfa to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08aaece98f3a469a9f14fa9841d50cfa 2023-04-16 20:54:33,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08a3d7fedebc4ceaaecf49b602e09b2c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/08a3d7fedebc4ceaaecf49b602e09b2c 2023-04-16 20:54:33,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d7e4b7ff4a9421395282bf64171486a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/0d7e4b7ff4a9421395282bf64171486a 2023-04-16 20:54:33,358 DEBUG [StoreCloser-TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/5b9dbba19bf749ca824636b7aaf260da to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/5b9dbba19bf749ca824636b7aaf260da 2023-04-16 20:54:33,362 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/recovered.edits/481.seqid, newMaxSeqId=481, maxSeqId=4 2023-04-16 20:54:33,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16. 2023-04-16 20:54:33,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fb9117f6cdea5099d0a91a955c183b16: 2023-04-16 20:54:33,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:33,365 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fb9117f6cdea5099d0a91a955c183b16, regionState=CLOSED 2023-04-16 20:54:33,365 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678473365"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678473365"}]},"ts":"1681678473365"} 2023-04-16 20:54:33,367 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-16 20:54:33,367 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure fb9117f6cdea5099d0a91a955c183b16, server=jenkins-hbase4.apache.org,41211,1681678416093 in 3.2410 sec 2023-04-16 20:54:33,368 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-16 20:54:33,368 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb9117f6cdea5099d0a91a955c183b16, UNASSIGN in 3.2440 sec 2023-04-16 20:54:33,369 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678473368"}]},"ts":"1681678473368"} 2023-04-16 20:54:33,369 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 20:54:33,371 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 20:54:33,372 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.2550 sec 2023-04-16 20:54:34,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 20:54:34,224 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-16 20:54:34,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 20:54:34,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:34,227 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:34,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-16 20:54:34,228 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:34,232 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:34,234 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/recovered.edits] 2023-04-16 20:54:34,236 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/0ea69b2c16564cce969d73f976416c8d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/0ea69b2c16564cce969d73f976416c8d 2023-04-16 20:54:34,237 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/98155403913f45fbb6ae1de04fae2e7a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/98155403913f45fbb6ae1de04fae2e7a 2023-04-16 20:54:34,238 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/c36f6711ca0645d7bb34f9886b6591a9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/c36f6711ca0645d7bb34f9886b6591a9 2023-04-16 20:54:34,240 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/dadd9009b7d74af29ab0cde675380276 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/dadd9009b7d74af29ab0cde675380276 2023-04-16 20:54:34,241 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e7e86c8e1b0a4cf3a2f199d23c884e47 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/A/e7e86c8e1b0a4cf3a2f199d23c884e47 2023-04-16 20:54:34,243 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/04ad39808b3f4c988d202c7623d7adc5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/04ad39808b3f4c988d202c7623d7adc5 2023-04-16 20:54:34,244 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/29e599e196a14634bf3f3a521ee82a81 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/29e599e196a14634bf3f3a521ee82a81 2023-04-16 20:54:34,245 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/33d6a82cb5b947608d4905bfa3eba036 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/33d6a82cb5b947608d4905bfa3eba036 2023-04-16 20:54:34,246 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/c03769b5afb648a6912cf9b8c479487a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/c03769b5afb648a6912cf9b8c479487a 2023-04-16 20:54:34,247 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/cb4e63f7a4d4457fa699fc6d9ac35820 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/B/cb4e63f7a4d4457fa699fc6d9ac35820 2023-04-16 20:54:34,249 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/292107dbb3284b39ae91e723699e0e62 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/292107dbb3284b39ae91e723699e0e62 2023-04-16 20:54:34,249 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/6fe14c038785405b83af6547d39d83c9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/6fe14c038785405b83af6547d39d83c9 2023-04-16 20:54:34,250 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/72d85d5ecc0047bd83da6db25c373a96 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/72d85d5ecc0047bd83da6db25c373a96 2023-04-16 20:54:34,251 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/93478278e8584fff8ef62d42379bcdfa to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/93478278e8584fff8ef62d42379bcdfa 2023-04-16 20:54:34,253 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fa60d522092a4380a2010cb4e4445384 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/C/fa60d522092a4380a2010cb4e4445384 2023-04-16 20:54:34,255 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/recovered.edits/481.seqid to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16/recovered.edits/481.seqid 2023-04-16 20:54:34,256 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/fb9117f6cdea5099d0a91a955c183b16 2023-04-16 20:54:34,256 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:54:34,256 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 20:54:34,257 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-16 20:54:34,261 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041603f1a38b85a94b5792ae7caf3aee7310 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041603f1a38b85a94b5792ae7caf3aee7310 2023-04-16 20:54:34,262 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160a694d47069444ba8a757068a111bf0e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160a694d47069444ba8a757068a111bf0e 2023-04-16 20:54:34,264 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161b0c0ef26519485fbbe5e9bff07103b5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161b0c0ef26519485fbbe5e9bff07103b5 2023-04-16 20:54:34,265 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161bc40d818b2d4814b7495369154d7171 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161bc40d818b2d4814b7495369154d7171 2023-04-16 20:54:34,266 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416254b26fe1a014265bc7fb5ad6a3939a8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416254b26fe1a014265bc7fb5ad6a3939a8 2023-04-16 20:54:34,267 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416261465b8fa024b348b69ac26b38a4135 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416261465b8fa024b348b69ac26b38a4135 2023-04-16 20:54:34,268 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416278a8cee54ac458b89bf0a12ca783ab7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416278a8cee54ac458b89bf0a12ca783ab7 2023-04-16 20:54:34,269 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164445faa1d080427cab3db813e5b2474d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164445faa1d080427cab3db813e5b2474d 2023-04-16 20:54:34,270 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164b107caef1c14f76907165d1b90c4e8b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164b107caef1c14f76907165d1b90c4e8b 2023-04-16 20:54:34,271 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041677f5e70b702b4cdd88815d85e40e4f1f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041677f5e70b702b4cdd88815d85e40e4f1f 2023-04-16 20:54:34,272 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167f1e1611d46846b990db22b44ec17840 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167f1e1611d46846b990db22b44ec17840 2023-04-16 20:54:34,274 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167f366193450941ae9385c6bb607407f1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167f366193450941ae9385c6bb607407f1 2023-04-16 20:54:34,275 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416810219fd19a74b0d9351f3369c08d83a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416810219fd19a74b0d9351f3369c08d83a 2023-04-16 20:54:34,276 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041684088be85b9540c791d4d150b39f3715 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041684088be85b9540c791d4d150b39f3715 2023-04-16 20:54:34,277 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169caa1c385dc3475386bf8338f7d22138 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169caa1c385dc3475386bf8338f7d22138 2023-04-16 20:54:34,279 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a7f04f88032c48a8a99ce5a32e322403 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a7f04f88032c48a8a99ce5a32e322403 2023-04-16 20:54:34,281 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b123b4e660ba48aca06165153bd3aacd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b123b4e660ba48aca06165153bd3aacd 2023-04-16 20:54:34,282 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bc1a9bfe526146759a098a0b09ed21fa to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bc1a9bfe526146759a098a0b09ed21fa 2023-04-16 20:54:34,283 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bc231370cad24d53b0f374c9f90c465a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bc231370cad24d53b0f374c9f90c465a 2023-04-16 20:54:34,285 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416cca6a56fc5e04e2d9d0309d7ec1bf688 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416cca6a56fc5e04e2d9d0309d7ec1bf688 2023-04-16 20:54:34,286 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416de3f767e13054778862e39ae8e9c704c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416de3f767e13054778862e39ae8e9c704c 2023-04-16 20:54:34,288 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416eaf5814916634d5aba5d22ff92347659 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416eaf5814916634d5aba5d22ff92347659 2023-04-16 20:54:34,289 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f1a8b89f593641df90382ca4fbd9370e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f1a8b89f593641df90382ca4fbd9370e 2023-04-16 20:54:34,290 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f1c79dbec47c4d849c143aa988683b31 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f1c79dbec47c4d849c143aa988683b31 2023-04-16 20:54:34,292 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f6935be04eda41048ca49ce4a8c495a7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f6935be04eda41048ca49ce4a8c495a7 2023-04-16 20:54:34,292 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 20:54:34,294 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:34,296 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 20:54:34,298 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 20:54:34,299 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:34,299 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 20:54:34,300 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681678474299"}]},"ts":"9223372036854775807"} 2023-04-16 20:54:34,301 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 20:54:34,301 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => fb9117f6cdea5099d0a91a955c183b16, NAME => 'TestAcidGuarantees,,1681678447809.fb9117f6cdea5099d0a91a955c183b16.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 20:54:34,301 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 20:54:34,301 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681678474301"}]},"ts":"9223372036854775807"} 2023-04-16 20:54:34,303 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 20:54:34,304 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:34,305 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 79 msec 2023-04-16 20:54:34,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-16 20:54:34,328 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-16 20:54:34,341 INFO [Listener at localhost/45179] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233 (was 234), OpenFileDescriptor=574 (was 588), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=413 (was 408) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=532 (was 706) 2023-04-16 20:54:34,351 INFO [Listener at localhost/45179] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=233, OpenFileDescriptor=574, MaxFileDescriptor=60000, SystemLoadAverage=413, ProcessCount=169, AvailableMemoryMB=532 2023-04-16 20:54:34,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 20:54:34,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:54:34,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:34,355 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 20:54:34,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-16 20:54:34,356 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 20:54:34,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 20:54:34,357 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:34,357 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda empty. 2023-04-16 20:54:34,358 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:34,358 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:54:34,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 20:54:34,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 20:54:34,766 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 20:54:34,768 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 968ac8675f340fdac83d49b0dce63dda, NAME => 'TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp 2023-04-16 20:54:34,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 20:54:35,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:54:35,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 968ac8675f340fdac83d49b0dce63dda, disabling compactions & flushes 2023-04-16 20:54:35,173 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:35,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:35,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. after waiting 0 ms 2023-04-16 20:54:35,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:35,173 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:35,173 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:35,175 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 20:54:35,176 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678475176"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678475176"}]},"ts":"1681678475176"} 2023-04-16 20:54:35,177 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 20:54:35,178 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 20:54:35,178 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678475178"}]},"ts":"1681678475178"} 2023-04-16 20:54:35,179 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 20:54:35,183 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=968ac8675f340fdac83d49b0dce63dda, ASSIGN}] 2023-04-16 20:54:35,185 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=968ac8675f340fdac83d49b0dce63dda, ASSIGN 2023-04-16 20:54:35,186 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=968ac8675f340fdac83d49b0dce63dda, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=false 2023-04-16 20:54:35,337 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=968ac8675f340fdac83d49b0dce63dda, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:35,337 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678475337"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678475337"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678475337"}]},"ts":"1681678475337"} 2023-04-16 20:54:35,338 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:54:35,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 20:54:35,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:35,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 968ac8675f340fdac83d49b0dce63dda, NAME => 'TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:54:35,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:35,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:54:35,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:35,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:35,494 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:35,495 DEBUG [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A 2023-04-16 20:54:35,496 DEBUG [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A 2023-04-16 20:54:35,496 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:54:35,496 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 968ac8675f340fdac83d49b0dce63dda columnFamilyName A 2023-04-16 20:54:35,497 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] regionserver.HStore(310): Store=968ac8675f340fdac83d49b0dce63dda/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:54:35,497 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:35,498 DEBUG [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B 2023-04-16 20:54:35,498 DEBUG [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B 2023-04-16 20:54:35,498 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:54:35,498 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 968ac8675f340fdac83d49b0dce63dda columnFamilyName B 2023-04-16 20:54:35,499 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] regionserver.HStore(310): Store=968ac8675f340fdac83d49b0dce63dda/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:54:35,499 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:35,500 DEBUG [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C 2023-04-16 20:54:35,500 DEBUG [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C 2023-04-16 20:54:35,500 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:54:35,500 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 968ac8675f340fdac83d49b0dce63dda columnFamilyName C 2023-04-16 20:54:35,500 INFO [StoreOpener-968ac8675f340fdac83d49b0dce63dda-1] regionserver.HStore(310): Store=968ac8675f340fdac83d49b0dce63dda/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:54:35,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:35,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:35,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:35,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:54:35,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:35,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 20:54:35,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 968ac8675f340fdac83d49b0dce63dda; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73181743, jitterRate=0.09049294888973236}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:54:35,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:35,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., pid=30, masterSystemTime=1681678475489 2023-04-16 20:54:35,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:35,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:35,508 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=968ac8675f340fdac83d49b0dce63dda, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:35,508 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678475508"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678475508"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678475508"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678475508"}]},"ts":"1681678475508"} 2023-04-16 20:54:35,510 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-16 20:54:35,510 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 in 171 msec 2023-04-16 20:54:35,511 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-16 20:54:35,511 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=968ac8675f340fdac83d49b0dce63dda, ASSIGN in 327 msec 2023-04-16 20:54:35,512 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 20:54:35,512 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678475512"}]},"ts":"1681678475512"} 2023-04-16 20:54:35,512 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 20:54:35,514 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 20:54:35,515 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-16 20:54:36,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 20:54:36,460 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-16 20:54:36,461 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,465 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,467 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,468 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45460, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,469 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 20:54:36,471 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 20:54:36,472 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,478 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,479 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,482 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,483 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,486 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,486 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,491 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,491 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x08cd7cb0 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,494 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@753156b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,495 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,502 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,502 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,506 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,506 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,510 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,510 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x2de064cb to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,513 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f8ff99f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,514 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x54a39be8 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:54:36,519 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a4b7b8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:54:36,523 DEBUG [hconnection-0x2efa8a09-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,523 DEBUG [hconnection-0x6e1fda74-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,523 DEBUG [hconnection-0x3fb0ead7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,524 DEBUG [hconnection-0x2335c845-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,524 DEBUG [hconnection-0x29778257-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,525 DEBUG [hconnection-0x255f8b6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,525 DEBUG [hconnection-0x50bf12f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,525 DEBUG [hconnection-0x411d3549-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,528 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45478, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,528 DEBUG [hconnection-0x3e9c6a08-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,528 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45474, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,528 DEBUG [hconnection-0x6713ef5e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:54:36,528 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45484, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,528 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,529 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45500, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,528 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45490, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:36,531 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,531 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,531 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45516, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,534 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45518, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:54:36,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:36,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:36,536 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:36,536 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:36,536 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:36,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:36,538 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:36,538 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:36,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:36,539 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:36,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:36,539 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:36,539 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:36,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:36,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:36,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:36,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:36,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:36,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:36,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:36,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:36,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:36,549 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:36,549 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:36,549 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:36,549 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:36,550 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-16 20:54:36,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:36,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:36,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:36,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:36,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:36,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:36,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:36,554 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:36,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678476559, max timestamp=1681678476568Num uniques 859; 2023-04-16 20:54:36,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678476559, max timestamp=1681678476568Num uniques 859; 2023-04-16 20:54:36,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:36,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678476559, max timestamp=1681678476568Num uniques 859; 2023-04-16 20:54:36,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:36,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:36,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678476598, max timestamp=1681678476605Num uniques 1018; 2023-04-16 20:54:36,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 20:54:36,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678476598, max timestamp=1681678476605Num uniques 1018; 2023-04-16 20:54:36,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-16 20:54:36,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678476598, max timestamp=1681678476605Num uniques 1018; 2023-04-16 20:54:36,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-16 20:54:36,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681678476613, max timestamp=1681678476628Num uniques 1027; 2023-04-16 20:54:36,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681678476613, max timestamp=1681678476628Num uniques 1009; 2023-04-16 20:54:36,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-16 20:54:36,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678476622, max timestamp=1681678476628Num uniques 1009; 2023-04-16 20:54:36,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-16 20:54:36,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10392, suffix heap size=132904,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=227 2023-04-16 20:54:36,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678476659, max timestamp=1681678476665Num uniques 1086; 2023-04-16 20:54:36,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-16 20:54:36,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678476659, max timestamp=1681678476665Num uniques 1086; 2023-04-16 20:54:36,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-16 20:54:36,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678476659, max timestamp=1681678476665Num uniques 1009; 2023-04-16 20:54:36,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-16 20:54:36,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:36,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678536686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:36,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11210, totalHeapSize=25720, min timestamp=1681678476674, max timestamp=1681678476694Num uniques 1045; 2023-04-16 20:54:36,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=11210, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-16 20:54:36,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1681678476674, max timestamp=1681678476694Num uniques 1045; 2023-04-16 20:54:36,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=9610, suffix heap size=136648,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=210 2023-04-16 20:54:36,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11210, totalHeapSize=25720, min timestamp=1681678476674, max timestamp=1681678476694Num uniques 1045; 2023-04-16 20:54:36,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=11210, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-16 20:54:36,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:36,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678536716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:36,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1681678476707, max timestamp=1681678476717Num uniques 1104; 2023-04-16 20:54:36,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1681678476707, max timestamp=1681678476717Num uniques 1104; 2023-04-16 20:54:36,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=11624, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-16 20:54:36,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=11624, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-16 20:54:36,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678476712, max timestamp=1681678476717Num uniques 1069; 2023-04-16 20:54:36,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=6870, suffix heap size=139144,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=150 2023-04-16 20:54:36,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681678476758, max timestamp=1681678476764Num uniques 1113; 2023-04-16 20:54:36,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681678476758, max timestamp=1681678476764Num uniques 1009; 2023-04-16 20:54:36,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-16 20:54:36,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681678476758, max timestamp=1681678476764Num uniques 1113; 2023-04-16 20:54:36,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7458, suffix heap size=132904,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=163 2023-04-16 20:54:36,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-16 20:54:36,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681678476773, max timestamp=1681678476781Num uniques 1022; 2023-04-16 20:54:36,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-16 20:54:36,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681678476773, max timestamp=1681678476781Num uniques 1022; 2023-04-16 20:54:36,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681678476773, max timestamp=1681678476781Num uniques 1022; 2023-04-16 20:54:36,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-16 20:54:36,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-16 20:54:36,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681678476791, max timestamp=1681678476797Num uniques 1031; 2023-04-16 20:54:36,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681678476791, max timestamp=1681678476797Num uniques 1031; 2023-04-16 20:54:36,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-16 20:54:36,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-16 20:54:36,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681678476791, max timestamp=1681678476797Num uniques 1031; 2023-04-16 20:54:36,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-16 20:54:36,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681678476803, max timestamp=1681678476831Num uniques 1040; 2023-04-16 20:54:36,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-16 20:54:36,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681678476803, max timestamp=1681678476831Num uniques 1040; 2023-04-16 20:54:36,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-16 20:54:36,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8695, totalHeapSize=20000, min timestamp=1681678476803, max timestamp=1681678476832Num uniques 1040; 2023-04-16 20:54:36,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47625, new segment data size=8695, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-16 20:54:36,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1681678476863, max timestamp=1681678476867Num uniques 1049; 2023-04-16 20:54:36,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-16 20:54:36,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1681678476863, max timestamp=1681678476867Num uniques 1099; 2023-04-16 20:54:36,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1681678476863, max timestamp=1681678476867Num uniques 1049; 2023-04-16 20:54:36,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50324, new segment data size=9104, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-16 20:54:36,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-16 20:54:36,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681678476877, max timestamp=1681678476881Num uniques 1058; 2023-04-16 20:54:36,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681678476877, max timestamp=1681678476881Num uniques 1058; 2023-04-16 20:54:36,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-16 20:54:36,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-16 20:54:36,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681678476877, max timestamp=1681678476881Num uniques 1058; 2023-04-16 20:54:36,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-16 20:54:36,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:36,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678536893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:36,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:36,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681678476887, max timestamp=1681678476894Num uniques 1117; 2023-04-16 20:54:36,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678536894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:36,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-04-16 20:54:36,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681678476887, max timestamp=1681678476894Num uniques 1117; 2023-04-16 20:54:36,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-04-16 20:54:36,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681678476887, max timestamp=1681678476894Num uniques 1117; 2023-04-16 20:54:36,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-04-16 20:54:36,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:36,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678536911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:36,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681678476907, max timestamp=1681678476911Num uniques 1126; 2023-04-16 20:54:36,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-16 20:54:36,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681678476907, max timestamp=1681678476911Num uniques 1126; 2023-04-16 20:54:36,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681678476907, max timestamp=1681678476911Num uniques 1126; 2023-04-16 20:54:36,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-16 20:54:36,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-16 20:54:36,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1681678476943, max timestamp=1681678476951Num uniques 1085; 2023-04-16 20:54:36,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1681678476943, max timestamp=1681678476951Num uniques 1085; 2023-04-16 20:54:36,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49680, new segment data size=8460, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-16 20:54:36,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1681678476943, max timestamp=1681678476951Num uniques 1085; 2023-04-16 20:54:36,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49680, new segment data size=8460, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-16 20:54:36,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49680, new segment data size=8460, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-16 20:54:36,966 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/04004f2546c847199d44f1d346e6aa19 2023-04-16 20:54:36,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681678476964, max timestamp=1681678476972Num uniques 1044; 2023-04-16 20:54:36,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-16 20:54:36,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681678476964, max timestamp=1681678476972Num uniques 1044; 2023-04-16 20:54:36,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-16 20:54:36,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681678476964, max timestamp=1681678476972Num uniques 1044; 2023-04-16 20:54:36,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-16 20:54:36,980 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/79aee9c23caf4a93a049657a363ccedc 2023-04-16 20:54:36,991 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/4f4f6cb2cb314c7cb74d343329c5031b 2023-04-16 20:54:36,995 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/04004f2546c847199d44f1d346e6aa19 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/04004f2546c847199d44f1d346e6aa19 2023-04-16 20:54:37,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681678476990, max timestamp=1681678476998Num uniques 1053; 2023-04-16 20:54:37,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-16 20:54:37,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681678476990, max timestamp=1681678476998Num uniques 1053; 2023-04-16 20:54:37,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681678476990, max timestamp=1681678476998Num uniques 1053; 2023-04-16 20:54:37,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-16 20:54:37,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-16 20:54:37,002 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/04004f2546c847199d44f1d346e6aa19, entries=100, sequenceid=8, filesize=9.3 K 2023-04-16 20:54:37,003 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/79aee9c23caf4a93a049657a363ccedc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/79aee9c23caf4a93a049657a363ccedc 2023-04-16 20:54:37,008 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/79aee9c23caf4a93a049657a363ccedc, entries=100, sequenceid=8, filesize=9.3 K 2023-04-16 20:54:37,008 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/4f4f6cb2cb314c7cb74d343329c5031b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4f4f6cb2cb314c7cb74d343329c5031b 2023-04-16 20:54:37,012 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4f4f6cb2cb314c7cb74d343329c5031b, entries=100, sequenceid=8, filesize=9.3 K 2023-04-16 20:54:37,013 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=114.05 KB/116790 for 968ac8675f340fdac83d49b0dce63dda in 463ms, sequenceid=8, compaction requested=false 2023-04-16 20:54:37,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:37,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:37,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:37,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:37,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:37,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:37,013 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=120.76 KB heapSize=341.02 KB 2023-04-16 20:54:37,013 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:37,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:37,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:37,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:37,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:37,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:37,015 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:37,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e7064c3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:37,016 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:37,017 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,017 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:37,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:37,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:37,018 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,018 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:37,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:37,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:37,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:37,021 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:37,021 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:37,021 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:37,021 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:37,021 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:37,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,025 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,025 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,027 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678537025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678537027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:37,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:37,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,031 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:37,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,031 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,031 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:37,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678537028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678537029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678537029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,132 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:37,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678537130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678537130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:37,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:37,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:37,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:37,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678537133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,135 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:37,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:37,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:37,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678537133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:37,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:37,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:37,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678537137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:37,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:37,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:37,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:37,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:37,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:37,139 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,139 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:37,139 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:37,139 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:37,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:37,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:37,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1069dc56[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:37,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:37,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:37,145 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,145 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:37,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:37,146 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:37,146 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,146 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,146 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:37,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:37,148 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:37,148 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:37,149 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:37,149 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:37,149 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:37,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,150 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,151 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,158 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,158 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,158 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,158 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:37,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:37,158 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,158 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:37,159 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:37,159 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:37,159 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,159 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,159 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,159 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,259 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:37,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:37,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:37,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:37,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:37,262 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:37,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:37,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:37,263 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,263 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,264 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,264 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,264 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,264 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:37,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:37,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:37,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:37,265 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:37,265 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,265 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:37,266 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,266 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:37,266 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:37,266 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:37,266 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:37,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:37,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3219ae2e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:37,269 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:37,270 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,270 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:37,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:37,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,273 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:37,273 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,273 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:37,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:37,275 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:37,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:37,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:37,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:37,275 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:37,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,280 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:37,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,282 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:37,283 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:37,283 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:37,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678537334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678537334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678537336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678537337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678537340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,383 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:37,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:37,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:37,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:37,387 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:37,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:37,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:37,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,388 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,388 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,389 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,389 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:37,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:37,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:37,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:37,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:37,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:37,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:37,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:37,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:37,390 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,390 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:37,390 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:37,390 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:37,391 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:37,391 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,391 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:37,391 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:37,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:37,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:37,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53bdb6bb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:37,394 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:37,394 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,395 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:37,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:37,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:37,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:37,397 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,397 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:37,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,397 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,397 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:37,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:37,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:37,399 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:37,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:37,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:37,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:37,399 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:37,400 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,400 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,400 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,401 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,402 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,405 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:37,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:37,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:37,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,405 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:37,405 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:37,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/163b3acdd2ba4cf7ad3da40d71863d50 2023-04-16 20:54:37,505 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:37,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:37,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:37,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:37,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:37,509 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:37,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:37,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:37,510 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,510 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,511 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,511 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,511 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,511 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:37,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:37,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:37,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:37,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:37,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:37,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:37,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:37,513 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,513 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:37,513 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:37,513 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:37,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:37,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:37,514 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:37,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:37,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec68673[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:37,517 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:37,517 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,518 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:37,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:37,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:37,518 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,518 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:37,520 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:37,520 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,520 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,520 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:37,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:37,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:37,523 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:37,523 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:37,523 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:37,523 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:37,523 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:37,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,525 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:37,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:37,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,530 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:37,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,530 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:37,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,630 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:37,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:37,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:37,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:37,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:37,634 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:37,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:37,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:37,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,637 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,637 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,637 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:37,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678537637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:37,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:37,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:37,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:37,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:37,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:37,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:37,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:37,639 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,639 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:37,639 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:37,639 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:37,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:37,640 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,640 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:37,640 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:37,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678537638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:37,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678537639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678537640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:37,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20937e91[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:37,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:37,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:37,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678537644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,650 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,650 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:37,650 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:37,650 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:37,651 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,651 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:37,651 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:37,651 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,652 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,652 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:37,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:37,655 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:37,655 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:37,655 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:37,655 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:37,655 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:37,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,658 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,660 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:37,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:37,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,669 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:37,669 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,769 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:37,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:37,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:37,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:37,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:37,773 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:37,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:37,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:37,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,776 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,776 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,776 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,776 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:37,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:37,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:37,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:37,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:37,778 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,778 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:37,778 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:37,778 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:37,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:37,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:37,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f04cc73[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:37,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,783 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:37,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:37,783 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:37,783 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:37,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:37,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:37,787 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:37,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:37,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:37,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:37,787 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:37,790 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,790 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,790 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,791 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,792 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,795 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:37,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:37,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:37,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,795 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:37,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,795 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:37,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/c79503620c0445048d46ed74673939fe 2023-04-16 20:54:37,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/158271bdb65547049b0cc26d05b0db7f 2023-04-16 20:54:37,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/163b3acdd2ba4cf7ad3da40d71863d50 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/163b3acdd2ba4cf7ad3da40d71863d50 2023-04-16 20:54:37,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/163b3acdd2ba4cf7ad3da40d71863d50, entries=200, sequenceid=317, filesize=14.3 K 2023-04-16 20:54:37,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/c79503620c0445048d46ed74673939fe as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c79503620c0445048d46ed74673939fe 2023-04-16 20:54:37,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c79503620c0445048d46ed74673939fe, entries=150, sequenceid=317, filesize=11.9 K 2023-04-16 20:54:37,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/158271bdb65547049b0cc26d05b0db7f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/158271bdb65547049b0cc26d05b0db7f 2023-04-16 20:54:37,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/158271bdb65547049b0cc26d05b0db7f, entries=150, sequenceid=317, filesize=11.9 K 2023-04-16 20:54:37,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.97 KB/349152, currentSize=60.38 KB/61830 for 968ac8675f340fdac83d49b0dce63dda in 861ms, sequenceid=317, compaction requested=false 2023-04-16 20:54:37,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:37,896 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:37,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:37,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:37,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:37,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:37,899 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:37,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:37,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:37,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,901 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:37,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:37,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:37,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:37,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:37,902 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:37,902 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:37,903 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:37,903 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:37,903 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:37,903 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,903 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:37,903 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:54:37,903 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:37,903 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:37,903 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:37,903 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:37,903 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:37,903 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:37,910 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/6421c4d15d5241969c29ba9d4130352b 2023-04-16 20:54:37,920 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-16 20:54:37,921 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/c133ff63ee11401290c6f1158b98dbdd 2023-04-16 20:54:37,931 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/71147480450c43d7aee0a3136c6c44b5 2023-04-16 20:54:37,935 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/6421c4d15d5241969c29ba9d4130352b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6421c4d15d5241969c29ba9d4130352b 2023-04-16 20:54:37,938 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6421c4d15d5241969c29ba9d4130352b, entries=150, sequenceid=329, filesize=11.9 K 2023-04-16 20:54:37,939 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/c133ff63ee11401290c6f1158b98dbdd as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c133ff63ee11401290c6f1158b98dbdd 2023-04-16 20:54:37,943 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c133ff63ee11401290c6f1158b98dbdd, entries=150, sequenceid=329, filesize=11.9 K 2023-04-16 20:54:37,943 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/71147480450c43d7aee0a3136c6c44b5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/71147480450c43d7aee0a3136c6c44b5 2023-04-16 20:54:37,947 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/71147480450c43d7aee0a3136c6c44b5, entries=150, sequenceid=329, filesize=11.9 K 2023-04-16 20:54:37,948 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 968ac8675f340fdac83d49b0dce63dda in 45ms, sequenceid=329, compaction requested=true 2023-04-16 20:54:37,948 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:37,948 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:37,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,949 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,950 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,950 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:37,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6132b146[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:37,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:37,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,954 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,954 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:37,954 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:37,954 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:37,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:37,956 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:37,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:37,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:37,959 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:37,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:37,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:37,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:37,959 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:37,961 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,961 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:37,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,961 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,961 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:37,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,963 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:37,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:37,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:37,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:37,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:37,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:37,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:37,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:37,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:37,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:37,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:37,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:37,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:37,967 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:37,968 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:38,068 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:38,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:38,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:38,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:38,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:38,071 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:38,071 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,071 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:38,071 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:38,072 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,072 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,074 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,074 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,074 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,074 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:38,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:38,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:38,076 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,076 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:38,076 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:38,076 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:38,077 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,077 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:38,077 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:38,080 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,080 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,080 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:38,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:38,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0d460d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:38,083 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:38,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:38,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:38,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:38,084 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:38,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:38,087 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:38,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:38,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:38,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:38,087 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:38,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:38,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:38,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,095 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:38,095 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:38,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:38,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:38,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:54:38,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:38,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:38,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:38,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:38,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:38,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:38,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678538163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678478159, max timestamp=1681678478166Num uniques 859; 2023-04-16 20:54:38,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:54:38,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678478159, max timestamp=1681678478166Num uniques 859; 2023-04-16 20:54:38,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:54:38,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678478159, max timestamp=1681678478166Num uniques 859; 2023-04-16 20:54:38,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:54:38,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678538178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678538181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678538182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678538183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,195 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:38,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:38,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:38,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:38,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:38,198 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:38,198 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,198 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:38,198 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:38,199 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,199 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,200 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,200 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,200 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,200 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:38,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:38,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:38,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:38,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:38,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:38,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:38,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:38,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:38,202 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,202 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:38,202 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:38,202 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:38,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:38,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:38,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:38,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,203 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,204 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,204 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:38,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d1d20b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:38,205 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:38,205 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,207 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:38,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:38,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:38,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:38,209 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:38,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:38,211 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:38,211 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:38,211 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:38,211 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:38,211 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:38,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,213 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:38,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:38,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:38,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,218 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:38,218 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:38,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678538269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678538284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678538286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678538286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678538286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,318 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:38,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:38,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:38,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:38,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:38,321 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:38,321 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,321 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:38,321 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:38,321 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,321 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,327 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:38,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:38,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:38,328 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:38,329 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,329 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:38,329 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:38,329 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:38,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:38,331 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,331 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,331 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,331 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:38,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@694ff336[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:38,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:38,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,334 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,334 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:38,334 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:38,334 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,334 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:38,334 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:38,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,336 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,336 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:38,339 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:38,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:38,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:38,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:38,339 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:38,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,341 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,342 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:38,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:38,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:38,344 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,345 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:38,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,345 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:38,445 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:38,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:38,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:38,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:38,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:38,448 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:38,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:38,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:38,449 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,449 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,450 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:38,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:38,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:38,452 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,452 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:38,452 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:38,452 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:38,455 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:38,456 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:38,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d83c7d8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:38,456 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,460 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:38,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:38,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:38,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:38,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:38,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,461 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:38,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:38,464 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:38,464 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:38,464 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:38,464 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:38,464 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:38,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,466 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,467 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:38,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:38,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,471 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,472 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:38,472 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,472 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:38,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678538470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678538485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678538489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678538489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678538489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/ea01a599220143258ff13ffa1c040cce 2023-04-16 20:54:38,572 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:38,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:38,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:38,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:38,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:38,584 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:38,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:38,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:38,585 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,585 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/d8d0e9bf352c4997a4957806a273ef4a 2023-04-16 20:54:38,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,586 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,586 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,586 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:38,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:38,588 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,588 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:38,588 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:38,588 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:38,590 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,590 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,591 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,591 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:38,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:38,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cffd74b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:38,593 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:38,593 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,594 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:38,594 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,594 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:38,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:38,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:38,600 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:38,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:38,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:38,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:38,600 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:38,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:38,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:38,603 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:38,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,604 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:38,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:38,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:38,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:38,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:38,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:38,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:38,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:38,607 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:38,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:38,608 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:38,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/2d7b79ead3ba416b8e8ab8b6b13d24fd 2023-04-16 20:54:38,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/ea01a599220143258ff13ffa1c040cce as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ea01a599220143258ff13ffa1c040cce 2023-04-16 20:54:38,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ea01a599220143258ff13ffa1c040cce, entries=200, sequenceid=342, filesize=14.3 K 2023-04-16 20:54:38,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/d8d0e9bf352c4997a4957806a273ef4a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/d8d0e9bf352c4997a4957806a273ef4a 2023-04-16 20:54:38,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/d8d0e9bf352c4997a4957806a273ef4a, entries=150, sequenceid=342, filesize=11.9 K 2023-04-16 20:54:38,628 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-16 20:54:38,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/2d7b79ead3ba416b8e8ab8b6b13d24fd as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d7b79ead3ba416b8e8ab8b6b13d24fd 2023-04-16 20:54:38,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d7b79ead3ba416b8e8ab8b6b13d24fd, entries=150, sequenceid=342, filesize=11.9 K 2023-04-16 20:54:38,633 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 968ac8675f340fdac83d49b0dce63dda in 484ms, sequenceid=342, compaction requested=true 2023-04-16 20:54:38,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:38,634 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:38,634 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:38,634 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:38,634 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:38,634 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:38,635 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50980 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:38,635 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:38,635 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/A is initiating minor compaction (all files) 2023-04-16 20:54:38,635 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/B is initiating minor compaction (all files) 2023-04-16 20:54:38,635 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/A in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,635 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/B in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,635 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/04004f2546c847199d44f1d346e6aa19, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/163b3acdd2ba4cf7ad3da40d71863d50, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6421c4d15d5241969c29ba9d4130352b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ea01a599220143258ff13ffa1c040cce] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=49.8 K 2023-04-16 20:54:38,635 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/79aee9c23caf4a93a049657a363ccedc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c79503620c0445048d46ed74673939fe, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c133ff63ee11401290c6f1158b98dbdd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/d8d0e9bf352c4997a4957806a273ef4a] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=45.0 K 2023-04-16 20:54:38,636 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 04004f2546c847199d44f1d346e6aa19, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681678476543 2023-04-16 20:54:38,636 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 79aee9c23caf4a93a049657a363ccedc, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681678476543 2023-04-16 20:54:38,636 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 163b3acdd2ba4cf7ad3da40d71863d50, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681678477006 2023-04-16 20:54:38,636 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c79503620c0445048d46ed74673939fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681678477006 2023-04-16 20:54:38,637 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6421c4d15d5241969c29ba9d4130352b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681678477022 2023-04-16 20:54:38,637 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c133ff63ee11401290c6f1158b98dbdd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681678477022 2023-04-16 20:54:38,637 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ea01a599220143258ff13ffa1c040cce, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681678478144 2023-04-16 20:54:38,637 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d8d0e9bf352c4997a4957806a273ef4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681678478144 2023-04-16 20:54:38,645 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#B#compaction#241 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:38,645 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#A#compaction#240 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:38,656 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/76329a41a2724fb2adddedc7e1e15d28 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/76329a41a2724fb2adddedc7e1e15d28 2023-04-16 20:54:38,662 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/B of 968ac8675f340fdac83d49b0dce63dda into 76329a41a2724fb2adddedc7e1e15d28(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:38,662 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:38,662 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/B, priority=12, startTime=1681678478634; duration=0sec 2023-04-16 20:54:38,662 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:38,662 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:38,663 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:38,663 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/C is initiating minor compaction (all files) 2023-04-16 20:54:38,664 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/C in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,664 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4f4f6cb2cb314c7cb74d343329c5031b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/158271bdb65547049b0cc26d05b0db7f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/71147480450c43d7aee0a3136c6c44b5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d7b79ead3ba416b8e8ab8b6b13d24fd] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=45.0 K 2023-04-16 20:54:38,664 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 4f4f6cb2cb314c7cb74d343329c5031b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681678476543 2023-04-16 20:54:38,664 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 158271bdb65547049b0cc26d05b0db7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681678477006 2023-04-16 20:54:38,665 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 71147480450c43d7aee0a3136c6c44b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681678477022 2023-04-16 20:54:38,665 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 2d7b79ead3ba416b8e8ab8b6b13d24fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681678478144 2023-04-16 20:54:38,672 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#C#compaction#242 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:38,708 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:38,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:38,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:38,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:38,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:38,711 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:38,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:38,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:38,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,713 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:38,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:38,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:38,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:38,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:38,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:38,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:38,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:38,714 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:38,714 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:38,714 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:38,715 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:38,715 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:38,715 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:38,715 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:38,715 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-16 20:54:38,715 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:38,715 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:38,716 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:38,716 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:38,716 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:38,716 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:38,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:38,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:38,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678538792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678538792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678538792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678538794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678538795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,893 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 20:54:38,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678538897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678538896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:38,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:38,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678538900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,056 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/870d35d40c434659978be1d36dce4caf as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/870d35d40c434659978be1d36dce4caf 2023-04-16 20:54:39,060 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/A of 968ac8675f340fdac83d49b0dce63dda into 870d35d40c434659978be1d36dce4caf(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:39,060 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:39,060 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/A, priority=12, startTime=1681678478633; duration=0sec 2023-04-16 20:54:39,061 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:39,091 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/e7aef42bfeff48799f352459cd732e46 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e7aef42bfeff48799f352459cd732e46 2023-04-16 20:54:39,096 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/C of 968ac8675f340fdac83d49b0dce63dda into e7aef42bfeff48799f352459cd732e46(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:39,096 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:39,096 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/C, priority=12, startTime=1681678478634; duration=0sec 2023-04-16 20:54:39,096 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:39,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678539101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678539101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678539108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,125 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/9fef556984b044a4959ce078aac3a64c 2023-04-16 20:54:39,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678539298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678539301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678539404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678539405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678539412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,535 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/71e9a6ec7a5741588c4078453b2ee869 2023-04-16 20:54:39,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678539907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678539908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:39,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678539914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,946 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/01cee1b27c2b4ebf96c89ce756d51baa 2023-04-16 20:54:39,950 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/9fef556984b044a4959ce078aac3a64c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/9fef556984b044a4959ce078aac3a64c 2023-04-16 20:54:39,954 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/9fef556984b044a4959ce078aac3a64c, entries=150, sequenceid=376, filesize=11.9 K 2023-04-16 20:54:39,954 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/71e9a6ec7a5741588c4078453b2ee869 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/71e9a6ec7a5741588c4078453b2ee869 2023-04-16 20:54:39,958 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/71e9a6ec7a5741588c4078453b2ee869, entries=150, sequenceid=376, filesize=11.9 K 2023-04-16 20:54:39,959 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/01cee1b27c2b4ebf96c89ce756d51baa as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/01cee1b27c2b4ebf96c89ce756d51baa 2023-04-16 20:54:39,962 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/01cee1b27c2b4ebf96c89ce756d51baa, entries=150, sequenceid=376, filesize=11.9 K 2023-04-16 20:54:39,963 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=60.38 KB/61830 for 968ac8675f340fdac83d49b0dce63dda in 1248ms, sequenceid=376, compaction requested=false 2023-04-16 20:54:39,963 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:39,963 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:39,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:39,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:39,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:39,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:39,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:39,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:39,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:39,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:39,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:39,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:39,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a01755[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:39,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:39,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:39,967 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:39,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:39,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:39,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:39,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:39,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:39,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:39,969 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:39,969 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:39,969 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:39,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:39,972 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:39,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:39,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:39,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:39,972 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:39,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:39,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:39,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:39,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:39,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,976 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:39,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:39,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:39,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:39,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:39,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:39,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:39,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:39,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:39,979 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:39,979 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:39,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:39,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:39,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:39,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:39,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,079 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:40,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:40,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:40,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:40,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:40,083 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:40,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:40,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:40,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:40,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:40,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:40,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:40,086 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,086 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:40,087 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:54:40,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:40,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:40,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:40,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:40,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:40,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:40,096 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/ee8f30cebf434b6dbaccb16d9cb7cd30 2023-04-16 20:54:40,107 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/9fc6e204e9454cd193104eeaed892e5f 2023-04-16 20:54:40,121 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/9f11e8f8aa0349468bf8c4886e75229e 2023-04-16 20:54:40,126 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/ee8f30cebf434b6dbaccb16d9cb7cd30 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ee8f30cebf434b6dbaccb16d9cb7cd30 2023-04-16 20:54:40,130 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ee8f30cebf434b6dbaccb16d9cb7cd30, entries=150, sequenceid=390, filesize=11.9 K 2023-04-16 20:54:40,131 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/9fc6e204e9454cd193104eeaed892e5f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9fc6e204e9454cd193104eeaed892e5f 2023-04-16 20:54:40,135 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9fc6e204e9454cd193104eeaed892e5f, entries=150, sequenceid=390, filesize=11.9 K 2023-04-16 20:54:40,136 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/9f11e8f8aa0349468bf8c4886e75229e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9f11e8f8aa0349468bf8c4886e75229e 2023-04-16 20:54:40,139 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9f11e8f8aa0349468bf8c4886e75229e, entries=150, sequenceid=390, filesize=11.9 K 2023-04-16 20:54:40,140 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 968ac8675f340fdac83d49b0dce63dda in 53ms, sequenceid=390, compaction requested=true 2023-04-16 20:54:40,140 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:40,140 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:40,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,142 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,142 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:40,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54608a74[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:40,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:40,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,147 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:40,148 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:40,148 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:40,148 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,148 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:40,151 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:40,151 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,151 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,151 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:40,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:40,154 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:40,154 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:40,154 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:40,154 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:40,154 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:40,156 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,156 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,156 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,156 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,158 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,160 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,160 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,160 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:40,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:40,160 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:40,160 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,160 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:40,161 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:40,161 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,161 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,161 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,161 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,161 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,261 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:40,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:40,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:40,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:40,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:40,264 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:40,264 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,264 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:40,264 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:40,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,267 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,267 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,267 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:40,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:40,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:40,269 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:40,269 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,269 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:40,269 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:40,269 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:40,269 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,269 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:40,269 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:40,269 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,270 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:40,270 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:40,270 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,270 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:40,270 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:40,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,271 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,272 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,272 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:40,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57de3a27[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:40,274 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:40,274 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,275 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:40,275 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:40,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:40,275 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:40,276 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:40,276 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,276 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:40,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:40,279 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:40,279 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:40,279 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:40,279 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:40,279 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:40,281 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,281 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,281 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,284 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,286 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,286 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,286 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,286 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,286 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:40,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:40,286 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,286 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,286 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:40,286 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:40,286 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,287 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,287 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:40,315 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:54:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:40,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/75843512ddd642109ab02585bae0f47c 2023-04-16 20:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678480352, max timestamp=1681678480362Num uniques 859; 2023-04-16 20:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678480352, max timestamp=1681678480362Num uniques 859; 2023-04-16 20:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678480352, max timestamp=1681678480362Num uniques 859; 2023-04-16 20:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:40,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:40,387 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:40,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:40,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:40,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:40,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:40,390 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:40,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:40,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:40,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,391 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,391 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,392 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,392 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:40,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:40,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:40,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:40,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:40,395 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,395 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:40,395 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:40,395 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:40,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678540391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:40,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:40,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678540392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,397 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7539fa18[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:40,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:40,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,400 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:40,401 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,401 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:40,402 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,402 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:40,402 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,402 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:40,404 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:40,404 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:40,404 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:40,404 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:40,404 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:40,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,406 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,407 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:40,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:40,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:40,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,410 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:40,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,410 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:40,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,411 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,411 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:40,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678540496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:40,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678540496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,511 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:40,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:40,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:40,514 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:40,514 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,514 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:40,514 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:40,514 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,514 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,516 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:40,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:40,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:40,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:40,518 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,518 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:40,518 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:40,518 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:40,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,519 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:40,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:40,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37614155[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:40,522 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,522 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:40,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:40,522 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:40,522 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:40,523 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:40,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:40,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:40,526 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:40,526 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:40,526 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:40,526 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:40,526 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:40,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,528 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:40,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:40,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,534 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,534 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,534 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:40,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,634 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:40,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:40,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:40,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:40,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:40,638 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:40,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:40,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:40,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,639 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,639 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,639 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,639 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:40,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:40,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:40,641 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,641 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:40,641 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:40,641 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:40,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:40,643 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:40,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67024ee7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:40,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:40,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,647 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:40,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:40,648 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,649 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:40,649 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,649 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:40,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:40,651 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:40,651 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:40,651 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:40,651 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:40,651 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:40,653 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,653 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,653 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,654 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,655 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,657 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:40,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:40,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,658 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:40,658 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:40,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:40,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45484 deadline: 1681678540700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:40,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678540700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/00c10d3205da4f1f9b66868da1f3fcfc 2023-04-16 20:54:40,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/29a9710ee5f246db84c2e55852c37984 2023-04-16 20:54:40,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/75843512ddd642109ab02585bae0f47c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/75843512ddd642109ab02585bae0f47c 2023-04-16 20:54:40,758 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:40,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:40,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/75843512ddd642109ab02585bae0f47c, entries=200, sequenceid=402, filesize=14.3 K 2023-04-16 20:54:40,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/00c10d3205da4f1f9b66868da1f3fcfc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/00c10d3205da4f1f9b66868da1f3fcfc 2023-04-16 20:54:40,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:40,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:40,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:40,762 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:40,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:40,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:40,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/00c10d3205da4f1f9b66868da1f3fcfc, entries=150, sequenceid=402, filesize=11.9 K 2023-04-16 20:54:40,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/29a9710ee5f246db84c2e55852c37984 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/29a9710ee5f246db84c2e55852c37984 2023-04-16 20:54:40,764 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:40,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:40,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:40,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:40,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:40,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:40,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:40,766 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,766 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:40,766 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:40,766 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:40,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:40,767 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,767 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:40,767 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:40,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/29a9710ee5f246db84c2e55852c37984, entries=150, sequenceid=402, filesize=11.9 K 2023-04-16 20:54:40,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 968ac8675f340fdac83d49b0dce63dda in 454ms, sequenceid=402, compaction requested=true 2023-04-16 20:54:40,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:40,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:40,769 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:40,769 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:40,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:40,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,770 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,770 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/A is initiating minor compaction (all files) 2023-04-16 20:54:40,770 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:40,771 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/A in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,771 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/B is initiating minor compaction (all files) 2023-04-16 20:54:40,771 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/870d35d40c434659978be1d36dce4caf, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/9fef556984b044a4959ce078aac3a64c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ee8f30cebf434b6dbaccb16d9cb7cd30, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/75843512ddd642109ab02585bae0f47c] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=50.1 K 2023-04-16 20:54:40,771 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/B in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,771 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/76329a41a2724fb2adddedc7e1e15d28, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/71e9a6ec7a5741588c4078453b2ee869, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9fc6e204e9454cd193104eeaed892e5f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/00c10d3205da4f1f9b66868da1f3fcfc] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=47.7 K 2023-04-16 20:54:40,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,771 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 870d35d40c434659978be1d36dce4caf, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681678478144 2023-04-16 20:54:40,771 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 76329a41a2724fb2adddedc7e1e15d28, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681678478144 2023-04-16 20:54:40,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:40,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41ab5f22[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:40,771 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:40,772 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,772 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 71e9a6ec7a5741588c4078453b2ee869, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1681678478180 2023-04-16 20:54:40,772 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9fef556984b044a4959ce078aac3a64c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1681678478180 2023-04-16 20:54:40,772 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9fc6e204e9454cd193104eeaed892e5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681678478787 2023-04-16 20:54:40,772 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ee8f30cebf434b6dbaccb16d9cb7cd30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681678478787 2023-04-16 20:54:40,772 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 00c10d3205da4f1f9b66868da1f3fcfc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681678480312 2023-04-16 20:54:40,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,773 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 75843512ddd642109ab02585bae0f47c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681678480309 2023-04-16 20:54:40,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:40,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:40,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:40,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:40,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:40,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,775 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:40,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:40,778 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:40,778 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:40,778 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:40,778 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:40,778 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:40,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:40,783 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#A#compaction#252 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:40,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:40,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,784 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#B#compaction#253 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:40,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:40,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:40,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:40,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:40,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:40,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:40,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,788 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:40,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,788 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:40,788 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:40,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:40,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:40,888 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:40,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:40,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:40,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:40,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:40,894 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:40,894 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,894 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:40,894 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:40,895 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,895 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:40,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:40,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:40,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:40,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:40,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:40,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:40,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:40,898 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:40,898 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:40,898 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:40,898 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:40,898 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:40,898 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 20:54:40,898 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:40,898 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:40,898 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:40,898 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:40,898 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:40,898 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:40,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:40,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:40,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:40,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678540936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:40,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:40,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678540937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:40,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678540937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678541003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678541005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678541043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678541043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678541043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,202 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/8c0e7d2086f7457b843aafdc58028de8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/8c0e7d2086f7457b843aafdc58028de8 2023-04-16 20:54:41,207 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/A of 968ac8675f340fdac83d49b0dce63dda into 8c0e7d2086f7457b843aafdc58028de8(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:41,207 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:41,208 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/A, priority=12, startTime=1681678480769; duration=0sec 2023-04-16 20:54:41,208 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:41,208 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:41,209 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/c20c47e7d4b54609a032aef576c1a672 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c20c47e7d4b54609a032aef576c1a672 2023-04-16 20:54:41,209 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:41,209 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/C is initiating minor compaction (all files) 2023-04-16 20:54:41,209 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/C in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:41,209 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e7aef42bfeff48799f352459cd732e46, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/01cee1b27c2b4ebf96c89ce756d51baa, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9f11e8f8aa0349468bf8c4886e75229e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/29a9710ee5f246db84c2e55852c37984] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=47.7 K 2023-04-16 20:54:41,210 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e7aef42bfeff48799f352459cd732e46, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681678478144 2023-04-16 20:54:41,211 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 01cee1b27c2b4ebf96c89ce756d51baa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1681678478180 2023-04-16 20:54:41,211 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9f11e8f8aa0349468bf8c4886e75229e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681678478787 2023-04-16 20:54:41,211 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 29a9710ee5f246db84c2e55852c37984, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681678480312 2023-04-16 20:54:41,213 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/B of 968ac8675f340fdac83d49b0dce63dda into c20c47e7d4b54609a032aef576c1a672(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:41,213 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:41,214 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/B, priority=12, startTime=1681678480769; duration=0sec 2023-04-16 20:54:41,214 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:41,219 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#C#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:41,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678541245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678541246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678541246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,310 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/c0e0b84d65c544f5ad233db429da00b5 2023-04-16 20:54:41,320 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/dd9a4f384ba64717b2fefc6df1a2119b 2023-04-16 20:54:41,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678541508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678541509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678541549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678541550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:41,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678541551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,627 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/3ffc58a8be4b4bd59d1a2d21d1914da6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/3ffc58a8be4b4bd59d1a2d21d1914da6 2023-04-16 20:54:41,631 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/C of 968ac8675f340fdac83d49b0dce63dda into 3ffc58a8be4b4bd59d1a2d21d1914da6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:41,631 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:41,631 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/C, priority=12, startTime=1681678480769; duration=0sec 2023-04-16 20:54:41,631 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:41,728 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/784866c024ab49cfbe1306f361e6860c 2023-04-16 20:54:41,736 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/c0e0b84d65c544f5ad233db429da00b5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c0e0b84d65c544f5ad233db429da00b5 2023-04-16 20:54:41,739 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c0e0b84d65c544f5ad233db429da00b5, entries=150, sequenceid=438, filesize=11.9 K 2023-04-16 20:54:41,740 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/dd9a4f384ba64717b2fefc6df1a2119b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/dd9a4f384ba64717b2fefc6df1a2119b 2023-04-16 20:54:41,744 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/dd9a4f384ba64717b2fefc6df1a2119b, entries=150, sequenceid=438, filesize=11.9 K 2023-04-16 20:54:41,744 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/784866c024ab49cfbe1306f361e6860c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/784866c024ab49cfbe1306f361e6860c 2023-04-16 20:54:41,749 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/784866c024ab49cfbe1306f361e6860c, entries=150, sequenceid=438, filesize=11.9 K 2023-04-16 20:54:41,750 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 968ac8675f340fdac83d49b0dce63dda in 852ms, sequenceid=438, compaction requested=false 2023-04-16 20:54:41,750 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:41,750 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:41,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:41,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:41,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:41,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:41,751 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:41,752 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:41,752 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:41,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:41,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b788435[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:41,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:41,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:41,755 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:41,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:41,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:41,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:41,757 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:41,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:41,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:41,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:41,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:41,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:41,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:41,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:41,760 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:41,760 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:41,760 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:41,760 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:41,760 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:41,761 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:41,761 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:41,761 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:41,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:41,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:41,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:41,763 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:41,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:41,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,769 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:41,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:41,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:41,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:41,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:41,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:41,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:41,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:41,769 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:41,769 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:41,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:41,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:41,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:41,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:41,869 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:41,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:41,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:41,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:41,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:41,873 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:41,873 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:41,873 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:41,873 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:41,873 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:41,873 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:41,874 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:41,874 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:41,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:41,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:41,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:41,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:41,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:41,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:41,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:41,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:41,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:41,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:41,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:41,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:41,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:41,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:41,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:41,876 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:41,876 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:41,876 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:54:41,877 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:41,877 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:41,877 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:41,877 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:41,877 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:41,877 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:42,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:42,057 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:42,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678542093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678482088, max timestamp=1681678482094Num uniques 859; 2023-04-16 20:54:42,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:54:42,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678482088, max timestamp=1681678482094Num uniques 859; 2023-04-16 20:54:42,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:54:42,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678482088, max timestamp=1681678482094Num uniques 859; 2023-04-16 20:54:42,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:54:42,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678542116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678542119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678542197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45498 deadline: 1681678542220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45516 deadline: 1681678542224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,285 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/10b35ba7e0904abfbcc9ae5e05e8d5e0 2023-04-16 20:54:42,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678542400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678542422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678542426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678542517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678542519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,702 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/8b11eda4d100431a906bab2bf40dacad 2023-04-16 20:54:42,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45474 deadline: 1681678542705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678542726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:42,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:42,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678542731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,111 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/99dc69272e6a43de9224da7bdb713e53 2023-04-16 20:54:43,115 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/10b35ba7e0904abfbcc9ae5e05e8d5e0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/10b35ba7e0904abfbcc9ae5e05e8d5e0 2023-04-16 20:54:43,120 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/10b35ba7e0904abfbcc9ae5e05e8d5e0, entries=150, sequenceid=453, filesize=11.9 K 2023-04-16 20:54:43,121 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/8b11eda4d100431a906bab2bf40dacad as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/8b11eda4d100431a906bab2bf40dacad 2023-04-16 20:54:43,124 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/8b11eda4d100431a906bab2bf40dacad, entries=150, sequenceid=453, filesize=11.9 K 2023-04-16 20:54:43,125 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/99dc69272e6a43de9224da7bdb713e53 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/99dc69272e6a43de9224da7bdb713e53 2023-04-16 20:54:43,128 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/99dc69272e6a43de9224da7bdb713e53, entries=150, sequenceid=453, filesize=11.9 K 2023-04-16 20:54:43,129 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 968ac8675f340fdac83d49b0dce63dda in 1253ms, sequenceid=453, compaction requested=true 2023-04-16 20:54:43,129 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:43,129 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:43,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:43,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:43,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:43,131 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:43,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:43,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ae09e6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:43,132 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:43,133 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,134 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:43,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:43,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:43,134 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,134 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:43,135 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:43,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:43,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:43,138 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:43,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:43,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:43,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:43,138 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:43,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:43,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:43,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,145 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,145 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:43,145 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:43,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,146 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:43,209 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-16 20:54:43,209 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:43,209 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:43,209 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:43,209 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:43,209 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:43,209 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:43,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:43,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678543232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:43,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678543233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:43,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678543235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,245 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:43,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:43,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:43,249 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:43,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:43,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:43,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,251 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:43,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:43,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:43,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:43,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:43,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:43,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:43,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:43,252 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,252 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:43,253 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:43,253 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:43,254 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@332e12a2[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:43,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:43,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:43,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:43,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:43,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:43,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:43,260 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:43,262 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:43,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:43,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:43,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:43,262 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:43,264 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,264 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,264 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,266 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,268 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:43,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:43,269 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,269 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:43,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:43,269 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:43,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:43,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678543335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:43,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678543335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,369 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:43,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:43,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:43,372 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:43,372 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,372 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:43,372 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:43,373 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,373 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,375 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,375 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,375 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,375 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:43,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:43,376 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,377 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:43,377 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:43,377 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:43,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:43,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:43,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:43,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,378 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@286709aa[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:43,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:43,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,381 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:43,381 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:43,381 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:43,381 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,381 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:43,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:43,382 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:43,385 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:43,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:43,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:43,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:43,386 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:43,388 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,388 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,388 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,389 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:43,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:43,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:43,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,392 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,392 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:43,393 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:43,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,493 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:43,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:43,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:43,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:43,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:43,496 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:43,496 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,496 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:43,496 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:43,496 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,496 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,505 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:43,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:43,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:43,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:43,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:43,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:43,514 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,514 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:43,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:43,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@178179bf[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:43,517 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:43,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:43,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:43,517 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,517 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:43,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:43,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,518 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:43,518 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:43,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:43,521 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:43,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:43,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:43,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:43,521 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:43,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,524 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,525 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:43,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:43,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,527 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:43,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:43,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,527 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:43,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678543539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:43,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678543539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,619 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/7736b31809fa4038ace35bc345a975ea 2023-04-16 20:54:43,628 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:43,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:43,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/c59a9d0fe26042e69791d4400cfbccf9 2023-04-16 20:54:43,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:43,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:43,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:43,631 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:43,631 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,631 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:43,631 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:43,632 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,632 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,633 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:43,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:43,635 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:43,635 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,635 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:43,635 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:43,635 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:43,635 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,635 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:43,635 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:43,636 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:43,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:43,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:43,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:43,637 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,637 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,638 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21d6b390[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:43,639 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:43,639 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:43,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:43,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:43,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:43,646 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:43,646 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,646 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,646 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:43,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:43,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:43,649 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:43,650 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:43,650 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:43,650 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:43,650 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:43,651 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,651 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,651 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,652 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,653 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:43,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:43,655 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,656 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:43,656 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,656 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,656 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,656 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,656 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,656 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:43,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,756 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:43,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:43,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:43,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:43,759 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:43,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:43,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:43,760 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,760 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,762 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:43,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:43,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:43,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:43,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:43,765 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,765 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:43,765 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:43,765 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:43,767 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,767 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,767 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,767 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:43,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@622e0e0a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:43,769 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:43,769 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,770 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,770 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:43,770 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:43,770 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:43,771 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,771 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:43,772 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:43,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,772 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,772 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:43,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:43,775 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:43,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:43,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:43,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:43,775 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:43,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,779 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,780 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:43,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:43,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,783 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,783 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,783 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,783 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,783 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:43,783 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:43,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:43,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678543842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:43,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678543842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,883 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:43,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:43,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:43,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:43,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:43,887 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:43,887 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,887 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:43,887 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:43,887 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,887 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,888 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,889 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,889 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:43,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:43,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:43,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:43,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:43,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:43,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:43,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:43,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:43,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:43,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:43,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:43,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:43,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:43,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:43,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,892 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,892 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,892 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:43,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@721471bc[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:43,894 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:43,894 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,896 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:43,896 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:43,896 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:43,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:43,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:43,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:43,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:43,899 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:43,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:43,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:43,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:43,900 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:43,901 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,901 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:43,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,901 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:43,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,903 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,905 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,905 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,905 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:43,905 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:43,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:43,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:43,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:43,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:43,905 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:43,905 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:43,906 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:43,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:43,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:43,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:43,906 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:43,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:44,006 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:44,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:44,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:44,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:44,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:44,009 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:44,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:44,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:44,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:44,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:44,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:44,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:44,013 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,013 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:44,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:44,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:44,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:44,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:44,014 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:44,014 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:44,014 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:44,014 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:44,016 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:44,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:44,017 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:44,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec4b2c0[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:44,017 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:44,018 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:44,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:44,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:44,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:44,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:44,019 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,019 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:44,020 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:44,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:44,022 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:44,022 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:44,022 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:44,022 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:44,022 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:44,025 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,025 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,025 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:44,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:44,025 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:44,026 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:44,028 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:44,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:44,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:44,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:44,029 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:44,029 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,029 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:44,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:44,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:44,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:44,029 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:44,029 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:44,029 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:44,029 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:44,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/e28b848c40ce44bd93d1f9aa7725f3d3 2023-04-16 20:54:44,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/7736b31809fa4038ace35bc345a975ea as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/7736b31809fa4038ace35bc345a975ea 2023-04-16 20:54:44,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/7736b31809fa4038ace35bc345a975ea, entries=150, sequenceid=488, filesize=11.9 K 2023-04-16 20:54:44,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/c59a9d0fe26042e69791d4400cfbccf9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c59a9d0fe26042e69791d4400cfbccf9 2023-04-16 20:54:44,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c59a9d0fe26042e69791d4400cfbccf9, entries=150, sequenceid=488, filesize=11.9 K 2023-04-16 20:54:44,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/e28b848c40ce44bd93d1f9aa7725f3d3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e28b848c40ce44bd93d1f9aa7725f3d3 2023-04-16 20:54:44,065 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e28b848c40ce44bd93d1f9aa7725f3d3, entries=150, sequenceid=488, filesize=11.9 K 2023-04-16 20:54:44,066 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 968ac8675f340fdac83d49b0dce63dda in 857ms, sequenceid=488, compaction requested=true 2023-04-16 20:54:44,066 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:44,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:44,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:44,066 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:44,066 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:44,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:44,067 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:44,067 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:44,067 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/B is initiating minor compaction (all files) 2023-04-16 20:54:44,067 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/A is initiating minor compaction (all files) 2023-04-16 20:54:44,067 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/B in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:44,068 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/A in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:44,068 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c20c47e7d4b54609a032aef576c1a672, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/dd9a4f384ba64717b2fefc6df1a2119b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/8b11eda4d100431a906bab2bf40dacad, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c59a9d0fe26042e69791d4400cfbccf9] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=47.9 K 2023-04-16 20:54:44,068 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/8c0e7d2086f7457b843aafdc58028de8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c0e0b84d65c544f5ad233db429da00b5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/10b35ba7e0904abfbcc9ae5e05e8d5e0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/7736b31809fa4038ace35bc345a975ea] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=47.9 K 2023-04-16 20:54:44,068 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c20c47e7d4b54609a032aef576c1a672, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681678480312 2023-04-16 20:54:44,068 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8c0e7d2086f7457b843aafdc58028de8, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681678480312 2023-04-16 20:54:44,069 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting dd9a4f384ba64717b2fefc6df1a2119b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1681678480384 2023-04-16 20:54:44,069 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c0e0b84d65c544f5ad233db429da00b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1681678480384 2023-04-16 20:54:44,069 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8b11eda4d100431a906bab2bf40dacad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681678480928 2023-04-16 20:54:44,069 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 10b35ba7e0904abfbcc9ae5e05e8d5e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681678480928 2023-04-16 20:54:44,069 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c59a9d0fe26042e69791d4400cfbccf9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681678482113 2023-04-16 20:54:44,070 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7736b31809fa4038ace35bc345a975ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681678482113 2023-04-16 20:54:44,086 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#B#compaction#264 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:44,089 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#A#compaction#265 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:44,129 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:44,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:44,132 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/5957dacf663f4fcc9d7945388510381b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5957dacf663f4fcc9d7945388510381b 2023-04-16 20:54:44,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:44,137 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/B of 968ac8675f340fdac83d49b0dce63dda into 5957dacf663f4fcc9d7945388510381b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:44,138 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:44,138 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/B, priority=12, startTime=1681678484066; duration=0sec 2023-04-16 20:54:44,138 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:44,138 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:44,139 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:44,139 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/C is initiating minor compaction (all files) 2023-04-16 20:54:44,139 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/C in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:44,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:44,139 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/3ffc58a8be4b4bd59d1a2d21d1914da6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/784866c024ab49cfbe1306f361e6860c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/99dc69272e6a43de9224da7bdb713e53, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e28b848c40ce44bd93d1f9aa7725f3d3] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=47.9 K 2023-04-16 20:54:44,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:44,140 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:44,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:44,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:44,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:44,140 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3ffc58a8be4b4bd59d1a2d21d1914da6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681678480312 2023-04-16 20:54:44,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:44,140 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 784866c024ab49cfbe1306f361e6860c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1681678480384 2023-04-16 20:54:44,141 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 99dc69272e6a43de9224da7bdb713e53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681678480928 2023-04-16 20:54:44,141 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e28b848c40ce44bd93d1f9aa7725f3d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681678482113 2023-04-16 20:54:44,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:44,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:44,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:44,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:44,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:44,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:44,143 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:44,143 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:44,144 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:54:44,144 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:44,144 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:44,144 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:44,144 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:44,144 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:44,144 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:44,177 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#C#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:44,204 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/9e9f5204425c48a5a13556beeb6e608e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9e9f5204425c48a5a13556beeb6e608e 2023-04-16 20:54:44,212 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/C of 968ac8675f340fdac83d49b0dce63dda into 9e9f5204425c48a5a13556beeb6e608e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:44,212 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:44,212 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/C, priority=12, startTime=1681678484066; duration=0sec 2023-04-16 20:54:44,212 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:44,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:44,244 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:44,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678484278, max timestamp=1681678484289Num uniques 859; 2023-04-16 20:54:44,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678484278, max timestamp=1681678484289Num uniques 859; 2023-04-16 20:54:44,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:44,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:44,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678484278, max timestamp=1681678484289Num uniques 859; 2023-04-16 20:54:44,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:44,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:44,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678544340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:44,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678544345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:44,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678544349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:44,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678544443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,531 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/a64c15e1bf6346cba86b8f720ec3eb9f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/a64c15e1bf6346cba86b8f720ec3eb9f 2023-04-16 20:54:44,535 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/A of 968ac8675f340fdac83d49b0dce63dda into a64c15e1bf6346cba86b8f720ec3eb9f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:44,536 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:44,536 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/A, priority=12, startTime=1681678484066; duration=0sec 2023-04-16 20:54:44,536 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:44,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:44,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678544534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:44,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678544534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,539 DEBUG [Thread-1754] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:44,539 DEBUG [Thread-1756] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:44,574 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/4526cfecd30d4a629b15023e9032f622 2023-04-16 20:54:44,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:44,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678544646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:44,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678544951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:44,984 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/15c3cb1ae5d94446b8569671c68c6991 2023-04-16 20:54:44,994 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/1402ecebf2d447a69143f0adcfea9449 2023-04-16 20:54:44,997 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/4526cfecd30d4a629b15023e9032f622 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4526cfecd30d4a629b15023e9032f622 2023-04-16 20:54:45,001 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4526cfecd30d4a629b15023e9032f622, entries=150, sequenceid=500, filesize=11.9 K 2023-04-16 20:54:45,002 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/15c3cb1ae5d94446b8569671c68c6991 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/15c3cb1ae5d94446b8569671c68c6991 2023-04-16 20:54:45,005 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/15c3cb1ae5d94446b8569671c68c6991, entries=150, sequenceid=500, filesize=11.9 K 2023-04-16 20:54:45,006 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/1402ecebf2d447a69143f0adcfea9449 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/1402ecebf2d447a69143f0adcfea9449 2023-04-16 20:54:45,014 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/1402ecebf2d447a69143f0adcfea9449, entries=150, sequenceid=500, filesize=11.9 K 2023-04-16 20:54:45,015 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 968ac8675f340fdac83d49b0dce63dda in 871ms, sequenceid=500, compaction requested=false 2023-04-16 20:54:45,015 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:45,015 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:45,017 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1ad6bb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:45,020 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:45,020 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,023 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:45,023 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,023 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:45,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:45,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:45,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:45,027 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:45,027 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:45,027 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:45,027 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:45,027 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:45,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:45,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:45,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:45,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:45,029 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:45,030 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,032 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:45,032 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:45,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:45,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:45,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,032 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:45,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:45,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:45,033 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:45,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:45,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:45,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:45,033 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:45,033 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:45,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:45,133 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:45,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:45,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:45,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:45,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:45,137 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:45,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:45,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:45,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:45,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:45,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:45,139 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:45,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:45,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:45,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:45,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:45,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:45,141 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:45,141 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:45,141 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 20:54:45,141 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:45,142 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:45,142 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:45,142 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:45,142 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:45,142 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:45,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:45,352 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:45,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:45,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678545380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:45,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678545380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:45,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678545454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:45,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678545485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:45,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678545485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,549 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/17eedce6a5264fbd88ca8a5fab8bfabe 2023-04-16 20:54:45,559 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/b70a2bdfb77b426d89a06ef273da1391 2023-04-16 20:54:45,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:45,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678545688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:45,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678545689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,971 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/7e33a6ece6ae440cbf7210413e9b6345 2023-04-16 20:54:45,975 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/17eedce6a5264fbd88ca8a5fab8bfabe as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/17eedce6a5264fbd88ca8a5fab8bfabe 2023-04-16 20:54:45,979 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/17eedce6a5264fbd88ca8a5fab8bfabe, entries=150, sequenceid=538, filesize=11.9 K 2023-04-16 20:54:45,980 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/b70a2bdfb77b426d89a06ef273da1391 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b70a2bdfb77b426d89a06ef273da1391 2023-04-16 20:54:45,983 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b70a2bdfb77b426d89a06ef273da1391, entries=150, sequenceid=538, filesize=11.9 K 2023-04-16 20:54:45,984 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/7e33a6ece6ae440cbf7210413e9b6345 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7e33a6ece6ae440cbf7210413e9b6345 2023-04-16 20:54:45,987 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7e33a6ece6ae440cbf7210413e9b6345, entries=150, sequenceid=538, filesize=11.9 K 2023-04-16 20:54:45,987 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 968ac8675f340fdac83d49b0dce63dda in 846ms, sequenceid=538, compaction requested=true 2023-04-16 20:54:45,988 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:45,988 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:45,989 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,989 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:45,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:45,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6856d9ac[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:45,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:45,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,993 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:45,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:45,993 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:45,993 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:45,993 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,993 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:45,993 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:45,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:54:45,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:45,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:45,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:45,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:45,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:45,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:45,999 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:46,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:46,002 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:46,002 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:46,002 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:46,002 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:46,002 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:46,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/1eca3a92663b47f6bceaba2078cb86a0 2023-04-16 20:54:46,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,004 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,004 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,004 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,009 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:46,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:46,009 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,009 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,009 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:46,010 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:46,009 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,010 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,010 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,010 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,010 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,012 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/22143937bcfa466e80bcde0629def12c 2023-04-16 20:54:46,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678546036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678546039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,110 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:46,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:46,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:46,113 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:46,113 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,113 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:46,113 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:46,113 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,113 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,114 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,114 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,114 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,114 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:46,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:46,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:46,116 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,116 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:46,116 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:46,116 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:46,119 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391edc8b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:46,121 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:46,121 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,122 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,122 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,122 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:46,122 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:46,122 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:46,122 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:46,124 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:46,126 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:46,126 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:46,126 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:46,126 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:46,126 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:46,127 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,127 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,127 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,129 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,132 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:46,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:46,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:46,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,133 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,133 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:46,133 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:46,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:46,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678546139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:46,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678546141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,233 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:46,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:46,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:46,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:46,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:46,236 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:46,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,237 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:46,237 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:46,237 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,237 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,238 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,238 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,238 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,238 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:46,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:46,240 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,240 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:46,240 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:46,240 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:46,241 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,242 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,242 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:46,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:46,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72872bda[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:46,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,244 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:46,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:46,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:46,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:46,246 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:46,246 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,246 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:46,249 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:46,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:46,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:46,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:46,249 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:46,251 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,251 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,251 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,253 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:46,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:46,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:46,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,256 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:46,256 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:46,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678546342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:46,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678546345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,356 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:46,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:46,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:46,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:46,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:46,359 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:46,359 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,359 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:46,359 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:46,359 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,359 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,360 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,360 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,360 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:46,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:46,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:46,361 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:46,361 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:46,362 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,362 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:46,362 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:46,362 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,365 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:46,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e84843f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:46,367 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:46,367 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,368 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:46,368 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,368 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:46,369 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,369 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,369 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,369 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:46,371 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:46,371 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:46,371 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:46,371 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:46,371 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:46,372 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,373 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,380 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,380 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:46,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:46,380 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,380 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,381 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:46,381 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:46,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:46,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/7a3a6fc02b684d20a514f50e504231e9 2023-04-16 20:54:46,431 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/1eca3a92663b47f6bceaba2078cb86a0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1eca3a92663b47f6bceaba2078cb86a0 2023-04-16 20:54:46,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1eca3a92663b47f6bceaba2078cb86a0, entries=200, sequenceid=551, filesize=14.3 K 2023-04-16 20:54:46,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/22143937bcfa466e80bcde0629def12c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/22143937bcfa466e80bcde0629def12c 2023-04-16 20:54:46,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/22143937bcfa466e80bcde0629def12c, entries=150, sequenceid=551, filesize=11.9 K 2023-04-16 20:54:46,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/7a3a6fc02b684d20a514f50e504231e9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7a3a6fc02b684d20a514f50e504231e9 2023-04-16 20:54:46,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7a3a6fc02b684d20a514f50e504231e9, entries=150, sequenceid=551, filesize=11.9 K 2023-04-16 20:54:46,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 968ac8675f340fdac83d49b0dce63dda in 451ms, sequenceid=551, compaction requested=true 2023-04-16 20:54:46,444 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:46,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:46,444 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:46,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:46,444 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:46,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:46,445 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:46,445 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/A is initiating minor compaction (all files) 2023-04-16 20:54:46,445 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:46,446 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/A in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,446 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/B is initiating minor compaction (all files) 2023-04-16 20:54:46,446 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/a64c15e1bf6346cba86b8f720ec3eb9f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4526cfecd30d4a629b15023e9032f622, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/17eedce6a5264fbd88ca8a5fab8bfabe, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1eca3a92663b47f6bceaba2078cb86a0] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=50.4 K 2023-04-16 20:54:46,446 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/B in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,446 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5957dacf663f4fcc9d7945388510381b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/15c3cb1ae5d94446b8569671c68c6991, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b70a2bdfb77b426d89a06ef273da1391, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/22143937bcfa466e80bcde0629def12c] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=48.0 K 2023-04-16 20:54:46,446 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting a64c15e1bf6346cba86b8f720ec3eb9f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681678482113 2023-04-16 20:54:46,446 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5957dacf663f4fcc9d7945388510381b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681678482113 2023-04-16 20:54:46,447 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 4526cfecd30d4a629b15023e9032f622, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1681678483214 2023-04-16 20:54:46,447 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 15c3cb1ae5d94446b8569671c68c6991, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1681678483214 2023-04-16 20:54:46,447 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 17eedce6a5264fbd88ca8a5fab8bfabe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1681678484322 2023-04-16 20:54:46,447 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b70a2bdfb77b426d89a06ef273da1391, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1681678484322 2023-04-16 20:54:46,447 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1eca3a92663b47f6bceaba2078cb86a0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681678485357 2023-04-16 20:54:46,447 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 22143937bcfa466e80bcde0629def12c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681678485357 2023-04-16 20:54:46,463 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#A#compaction#276 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:46,463 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#B#compaction#277 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:46,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:46,467 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-16 20:54:46,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678486017, max timestamp=1681678486465Num uniques 859; 2023-04-16 20:54:46,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:46,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:46,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678486017, max timestamp=1681678486465Num uniques 859; 2023-04-16 20:54:46,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678486017, max timestamp=1681678486465Num uniques 859; 2023-04-16 20:54:46,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:46,468 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:46,468 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:46,468 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:46,468 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:46,468 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:46,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:46,481 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:46,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:46,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:46,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:46,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:46,492 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:46,492 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,492 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:46,492 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:46,493 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,493 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,494 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:46,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:46,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:46,496 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,496 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:46,497 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:46,497 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:46,498 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/2012de31779049308a986b85a462b1ab as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/2012de31779049308a986b85a462b1ab 2023-04-16 20:54:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,498 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:46,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c1bdfa[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:46,501 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:46,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,503 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,503 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:46,503 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:46,503 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:46,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:46,503 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/A of 968ac8675f340fdac83d49b0dce63dda into 2012de31779049308a986b85a462b1ab(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:46,503 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:46,503 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/A, priority=12, startTime=1681678486444; duration=0sec 2023-04-16 20:54:46,504 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:46,504 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:46,505 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,505 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/C is initiating minor compaction (all files) 2023-04-16 20:54:46,505 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,506 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/C in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:46,506 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9e9f5204425c48a5a13556beeb6e608e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/1402ecebf2d447a69143f0adcfea9449, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7e33a6ece6ae440cbf7210413e9b6345, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7a3a6fc02b684d20a514f50e504231e9] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=48.0 K 2023-04-16 20:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,506 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9e9f5204425c48a5a13556beeb6e608e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681678482113 2023-04-16 20:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,506 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1402ecebf2d447a69143f0adcfea9449, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1681678483214 2023-04-16 20:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,507 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7e33a6ece6ae440cbf7210413e9b6345, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1681678484322 2023-04-16 20:54:46,507 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7a3a6fc02b684d20a514f50e504231e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681678485357 2023-04-16 20:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:46,509 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:46,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:46,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:46,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:46,509 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:46,511 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,511 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,511 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,512 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,513 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:46,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:46,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,516 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,516 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,516 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:46,516 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:46,516 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#C#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:46,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:46,530 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/8a51eb8ab7b94d16a5008f8127da78b0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/8a51eb8ab7b94d16a5008f8127da78b0 2023-04-16 20:54:46,534 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/C of 968ac8675f340fdac83d49b0dce63dda into 8a51eb8ab7b94d16a5008f8127da78b0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:46,534 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:46,534 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/C, priority=12, startTime=1681678486444; duration=0sec 2023-04-16 20:54:46,534 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:46,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678486531, max timestamp=1681678486542Num uniques 859; 2023-04-16 20:54:46,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678486531, max timestamp=1681678486542Num uniques 859; 2023-04-16 20:54:46,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:46,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:46,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678486531, max timestamp=1681678486542Num uniques 859; 2023-04-16 20:54:46,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:46,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678486579, max timestamp=1681678486596Num uniques 1018; 2023-04-16 20:54:46,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678486579, max timestamp=1681678486596Num uniques 1018; 2023-04-16 20:54:46,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 20:54:46,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 20:54:46,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678486579, max timestamp=1681678486596Num uniques 1018; 2023-04-16 20:54:46,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 20:54:46,616 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:46,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:46,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:46,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:46,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:46,619 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:46,619 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,619 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:46,619 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:46,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,621 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,621 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,621 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,621 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:46,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:46,623 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,623 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:46,623 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:46,623 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,625 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f74c3ed[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:46,627 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:46,627 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,629 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:46,629 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,629 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,630 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:46,633 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:46,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:46,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:46,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:46,633 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:46,635 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,635 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,635 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,637 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,640 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:46,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:46,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:46,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,641 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:46,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,641 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:46,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681678486640, max timestamp=1681678486651Num uniques 1027; 2023-04-16 20:54:46,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681678486644, max timestamp=1681678486651Num uniques 1027; 2023-04-16 20:54:46,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-16 20:54:46,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-16 20:54:46,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681678486644, max timestamp=1681678486651Num uniques 1027; 2023-04-16 20:54:46,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-16 20:54:46,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681678486666, max timestamp=1681678486673Num uniques 1036; 2023-04-16 20:54:46,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-16 20:54:46,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681678486666, max timestamp=1681678486673Num uniques 1036; 2023-04-16 20:54:46,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-16 20:54:46,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681678486666, max timestamp=1681678486673Num uniques 1086; 2023-04-16 20:54:46,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-16 20:54:46,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681678486687, max timestamp=1681678486692Num uniques 1095; 2023-04-16 20:54:46,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-16 20:54:46,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681678486687, max timestamp=1681678486692Num uniques 1095; 2023-04-16 20:54:46,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681678486687, max timestamp=1681678486692Num uniques 1095; 2023-04-16 20:54:46,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-16 20:54:46,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-16 20:54:46,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681678486704, max timestamp=1681678486716Num uniques 1054; 2023-04-16 20:54:46,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-16 20:54:46,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681678486704, max timestamp=1681678486716Num uniques 1054; 2023-04-16 20:54:46,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681678486704, max timestamp=1681678486716Num uniques 1054; 2023-04-16 20:54:46,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-16 20:54:46,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-16 20:54:46,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681678486724, max timestamp=1681678486735Num uniques 1013; 2023-04-16 20:54:46,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681678486724, max timestamp=1681678486735Num uniques 1013; 2023-04-16 20:54:46,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-16 20:54:46,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-16 20:54:46,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681678486724, max timestamp=1681678486735Num uniques 1013; 2023-04-16 20:54:46,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-16 20:54:46,741 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:46,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:46,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:46,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:46,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:46,752 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:46,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:46,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:46,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,754 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:46,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:46,756 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,756 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:46,756 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:46,756 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:46,757 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:46,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e6e822[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:46,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,760 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:46,760 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,760 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:46,763 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:46,765 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:46,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:46,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:46,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:46,765 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:46,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,766 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,767 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,768 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,772 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:46,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:46,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:46,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,773 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:46,773 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:46,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681678486749, max timestamp=1681678486784Num uniques 1122; 2023-04-16 20:54:46,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-16 20:54:46,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681678486749, max timestamp=1681678486784Num uniques 1122; 2023-04-16 20:54:46,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-16 20:54:46,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681678486749, max timestamp=1681678486784Num uniques 1122; 2023-04-16 20:54:46,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-16 20:54:46,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681678486794, max timestamp=1681678486803Num uniques 1081; 2023-04-16 20:54:46,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-16 20:54:46,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681678486794, max timestamp=1681678486803Num uniques 1081; 2023-04-16 20:54:46,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-16 20:54:46,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681678486794, max timestamp=1681678486803Num uniques 1081; 2023-04-16 20:54:46,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-16 20:54:46,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681678486824, max timestamp=1681678486848Num uniques 1090; 2023-04-16 20:54:46,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-16 20:54:46,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681678486824, max timestamp=1681678486848Num uniques 1090; 2023-04-16 20:54:46,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8692, totalHeapSize=20000, min timestamp=1681678486824, max timestamp=1681678486848Num uniques 1090; 2023-04-16 20:54:46,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-16 20:54:46,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49912, new segment data size=8692, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-16 20:54:46,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681678486860, max timestamp=1681678486868Num uniques 1049; 2023-04-16 20:54:46,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-16 20:54:46,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681678486860, max timestamp=1681678486868Num uniques 1049; 2023-04-16 20:54:46,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-16 20:54:46,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681678486860, max timestamp=1681678486868Num uniques 1049; 2023-04-16 20:54:46,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-16 20:54:46,873 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:46,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:46,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:46,876 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:46,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:46,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:46,877 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,877 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,879 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,879 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,879 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:46,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:46,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:46,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:46,881 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,881 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:46,881 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:46,881 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:46,882 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:46,885 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:46,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb931fd[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:46,885 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,886 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,886 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/3785df75078e40e09fbd087796fbe436 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3785df75078e40e09fbd087796fbe436 2023-04-16 20:54:46,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:46,886 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,886 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:46,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:46,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:46,890 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:46,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:46,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:46,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:46,890 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:46,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:46,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:46,893 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,894 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,896 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/B of 968ac8675f340fdac83d49b0dce63dda into 3785df75078e40e09fbd087796fbe436(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:46,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:46,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:46,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:46,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:46,896 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:46,896 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/B, priority=12, startTime=1681678486444; duration=0sec 2023-04-16 20:54:46,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:46,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:46,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:46,897 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:46,897 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:46,897 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:46,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:46,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:46,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:46,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:46,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:46,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:46,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678546908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678546907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:46,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681678486901, max timestamp=1681678486910Num uniques 1158; 2023-04-16 20:54:46,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681678486901, max timestamp=1681678486910Num uniques 1158; 2023-04-16 20:54:46,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-16 20:54:46,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-16 20:54:46,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681678486901, max timestamp=1681678486910Num uniques 1158; 2023-04-16 20:54:46,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-16 20:54:46,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/1ad66a24220b4e4b895d7ce7f35a24f3 2023-04-16 20:54:46,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681678486947, max timestamp=1681678486972Num uniques 1017; 2023-04-16 20:54:46,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-16 20:54:46,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681678486947, max timestamp=1681678486972Num uniques 1017; 2023-04-16 20:54:46,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-16 20:54:46,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681678486947, max timestamp=1681678486972Num uniques 1017; 2023-04-16 20:54:46,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-16 20:54:46,997 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:46,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:46,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:47,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:47,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:47,001 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:47,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:47,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:47,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,003 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,003 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,003 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,003 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:47,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:47,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:47,004 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:47,004 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,004 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:47,005 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,005 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:47,005 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:47,005 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:47,007 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:47,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739a6dbd[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:47,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:47,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,011 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:47,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:47,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:47,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:47,014 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:47,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,014 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1681678486967, max timestamp=1681678487014Num uniques 1026; 2023-04-16 20:54:47,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46976, new segment data size=8046, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-16 20:54:47,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681678486967, max timestamp=1681678487014Num uniques 1026; 2023-04-16 20:54:47,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-16 20:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8053, totalHeapSize=18544, min timestamp=1681678486967, max timestamp=1681678487014Num uniques 1026; 2023-04-16 20:54:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46983, new segment data size=8053, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-16 20:54:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:47,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:47,018 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:47,018 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:47,018 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:47,018 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:47,018 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:47,020 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,020 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,020 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,021 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,022 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:47,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:47,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:47,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,025 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:47,025 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:47,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681678487027, max timestamp=1681678487033Num uniques 1035; 2023-04-16 20:54:47,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-16 20:54:47,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681678487027, max timestamp=1681678487033Num uniques 1035; 2023-04-16 20:54:47,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-16 20:54:47,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681678487027, max timestamp=1681678487033Num uniques 1035; 2023-04-16 20:54:47,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-16 20:54:47,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681678487042, max timestamp=1681678487052Num uniques 1144; 2023-04-16 20:54:47,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=13464, suffix heap size=146944,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=294 2023-04-16 20:54:47,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681678487042, max timestamp=1681678487052Num uniques 1144; 2023-04-16 20:54:47,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=13464, suffix heap size=146944,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=294 2023-04-16 20:54:47,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681678487042, max timestamp=1681678487052Num uniques 1144; 2023-04-16 20:54:47,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=13464, suffix heap size=146944,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=294 2023-04-16 20:54:47,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:47,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678547071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681678487061, max timestamp=1681678487074Num uniques 1153; 2023-04-16 20:54:47,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681678487061, max timestamp=1681678487074Num uniques 1153; 2023-04-16 20:54:47,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=9295, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-04-16 20:54:47,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1681678487061, max timestamp=1681678487074Num uniques 1153; 2023-04-16 20:54:47,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=9295, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-04-16 20:54:47,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52798, new segment data size=9288, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-04-16 20:54:47,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681678487095, max timestamp=1681678487101Num uniques 1062; 2023-04-16 20:54:47,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-04-16 20:54:47,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681678487095, max timestamp=1681678487101Num uniques 1062; 2023-04-16 20:54:47,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-04-16 20:54:47,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681678487095, max timestamp=1681678487101Num uniques 1062; 2023-04-16 20:54:47,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-04-16 20:54:47,125 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:47,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:47,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:47,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:47,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:47,128 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:47,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:47,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:47,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,129 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:47,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:47,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:47,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:47,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:47,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:47,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:47,131 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,131 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:47,131 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:47,131 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:47,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:47,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:47,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681678487120, max timestamp=1681678487132Num uniques 1071; 2023-04-16 20:54:47,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-16 20:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:47,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681678487120, max timestamp=1681678487132Num uniques 1071; 2023-04-16 20:54:47,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-16 20:54:47,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681678487120, max timestamp=1681678487132Num uniques 1071; 2023-04-16 20:54:47,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-16 20:54:47,135 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:47,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a18b388[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:47,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:47,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,139 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:47,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,141 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:47,143 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:47,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:47,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:47,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:47,143 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:47,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,145 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,146 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,149 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:47,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:47,150 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,150 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,150 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:47,150 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:47,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:47,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8239, totalHeapSize=18960, min timestamp=1681678487154, max timestamp=1681678487165Num uniques 1080; 2023-04-16 20:54:47,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8233, totalHeapSize=18960, min timestamp=1681678487154, max timestamp=1681678487165Num uniques 1080; 2023-04-16 20:54:47,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681678487154, max timestamp=1681678487160Num uniques 1080; 2023-04-16 20:54:47,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49453, new segment data size=8233, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-16 20:54:47,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49459, new segment data size=8239, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-16 20:54:47,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-16 20:54:47,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681678487193, max timestamp=1681678487229Num uniques 1039; 2023-04-16 20:54:47,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-16 20:54:47,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681678487193, max timestamp=1681678487229Num uniques 1039; 2023-04-16 20:54:47,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-16 20:54:47,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681678487193, max timestamp=1681678487229Num uniques 1039; 2023-04-16 20:54:47,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-16 20:54:47,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681678487241, max timestamp=1681678487248Num uniques 1098; 2023-04-16 20:54:47,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-16 20:54:47,250 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:47,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681678487241, max timestamp=1681678487248Num uniques 1098; 2023-04-16 20:54:47,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681678487241, max timestamp=1681678487248Num uniques 1098; 2023-04-16 20:54:47,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-16 20:54:47,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:47,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-16 20:54:47,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:47,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:47,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:47,254 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:47,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:47,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:47,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:47,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:47,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:47,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:47,260 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,260 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:47,260 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:47,260 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:47,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:47,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:47,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@298cef11[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:47,264 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:47,264 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,265 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,265 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:47,266 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,266 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:47,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:47,270 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:47,270 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:47,270 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:47,270 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:47,270 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:47,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681678487257, max timestamp=1681678487270Num uniques 1107; 2023-04-16 20:54:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681678487257, max timestamp=1681678487270Num uniques 1107; 2023-04-16 20:54:47,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-04-16 20:54:47,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-04-16 20:54:47,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681678487257, max timestamp=1681678487270Num uniques 1107; 2023-04-16 20:54:47,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=7185, suffix heap size=143096,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=157 2023-04-16 20:54:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,274 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,278 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,278 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,278 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,278 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,278 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:47,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:47,278 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,278 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,278 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:47,278 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:47,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1681678487278, max timestamp=1681678487293Num uniques 1116; 2023-04-16 20:54:47,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1681678487278, max timestamp=1681678487293Num uniques 1116; 2023-04-16 20:54:47,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=9886, suffix heap size=144032,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=216 2023-04-16 20:54:47,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=9886, suffix heap size=144032,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=216 2023-04-16 20:54:47,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1681678487278, max timestamp=1681678487293Num uniques 1016; 2023-04-16 20:54:47,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=9886, suffix heap size=133632,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=216 2023-04-16 20:54:47,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681678487303, max timestamp=1681678487313Num uniques 1075; 2023-04-16 20:54:47,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-16 20:54:47,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681678487303, max timestamp=1681678487313Num uniques 1075; 2023-04-16 20:54:47,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681678487303, max timestamp=1681678487313Num uniques 1075; 2023-04-16 20:54:47,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-16 20:54:47,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-16 20:54:47,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1681678487327, max timestamp=1681678487332Num uniques 1084; 2023-04-16 20:54:47,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-16 20:54:47,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1681678487327, max timestamp=1681678487332Num uniques 1084; 2023-04-16 20:54:47,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-16 20:54:47,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1681678487327, max timestamp=1681678487332Num uniques 1084; 2023-04-16 20:54:47,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-16 20:54:47,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/54bef9c99b7d432dbec41c413c0a7d2d 2023-04-16 20:54:47,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1681678487338, max timestamp=1681678487356Num uniques 1043; 2023-04-16 20:54:47,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47758, new segment data size=11118, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-16 20:54:47,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1681678487338, max timestamp=1681678487356Num uniques 1043; 2023-04-16 20:54:47,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1681678487338, max timestamp=1681678487356Num uniques 1043; 2023-04-16 20:54:47,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47758, new segment data size=11118, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-16 20:54:47,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47758, new segment data size=9104, suffix heap size=136440,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=199 2023-04-16 20:54:47,378 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:47,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9242, totalHeapSize=21248, min timestamp=1681678487371, max timestamp=1681678487379Num uniques 1102; 2023-04-16 20:54:47,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50462, new segment data size=9242, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-16 20:54:47,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9242, totalHeapSize=21248, min timestamp=1681678487371, max timestamp=1681678487379Num uniques 1102; 2023-04-16 20:54:47,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50462, new segment data size=9242, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-16 20:54:47,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9242, totalHeapSize=21248, min timestamp=1681678487371, max timestamp=1681678487379Num uniques 1058; 2023-04-16 20:54:47,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9242, suffix heap size=138000,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=202 2023-04-16 20:54:47,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:47,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:47,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:47,385 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:47,385 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,385 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:47,385 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:47,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,388 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,389 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,389 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,389 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:47,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:47,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:47,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:47,392 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,392 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:47,393 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:47,393 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:47,395 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8d0504[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:47,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:47,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,399 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:47,399 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:47,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:47,399 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:47,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:47,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:47,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:47,403 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:47,403 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:47,403 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:47,403 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:47,403 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:47,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,407 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681678487393, max timestamp=1681678487407Num uniques 1061; 2023-04-16 20:54:47,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-16 20:54:47,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681678487393, max timestamp=1681678487407Num uniques 1061; 2023-04-16 20:54:47,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-16 20:54:47,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681678487393, max timestamp=1681678487407Num uniques 1061; 2023-04-16 20:54:47,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-04-16 20:54:47,411 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,414 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,414 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,414 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:47,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:47,414 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,414 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,414 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,414 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,414 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:47,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,414 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,415 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:47,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1681678487426, max timestamp=1681678487431Num uniques 1020; 2023-04-16 20:54:47,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-04-16 20:54:47,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1681678487426, max timestamp=1681678487431Num uniques 1020; 2023-04-16 20:54:47,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-04-16 20:54:47,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1681678487426, max timestamp=1681678487431Num uniques 1020; 2023-04-16 20:54:47,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-04-16 20:54:47,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681678487448, max timestamp=1681678487453Num uniques 1079; 2023-04-16 20:54:47,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-16 20:54:47,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681678487448, max timestamp=1681678487453Num uniques 1079; 2023-04-16 20:54:47,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681678487448, max timestamp=1681678487453Num uniques 1079; 2023-04-16 20:54:47,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-16 20:54:47,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-16 20:54:47,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10894, totalHeapSize=24992, min timestamp=1681678487466, max timestamp=1681678487475Num uniques 1038; 2023-04-16 20:54:47,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10888, totalHeapSize=24992, min timestamp=1681678487466, max timestamp=1681678487475Num uniques 1038; 2023-04-16 20:54:47,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681678487466, max timestamp=1681678487475Num uniques 1038; 2023-04-16 20:54:47,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47528, new segment data size=10888, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-16 20:54:47,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47534, new segment data size=10894, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-16 20:54:47,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-16 20:54:47,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1681678487488, max timestamp=1681678487495Num uniques 1097; 2023-04-16 20:54:47,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=9022, suffix heap size=142056,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=197 2023-04-16 20:54:47,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1681678487488, max timestamp=1681678487495Num uniques 1097; 2023-04-16 20:54:47,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=9022, suffix heap size=142056,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=197 2023-04-16 20:54:47,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1681678487488, max timestamp=1681678487495Num uniques 1097; 2023-04-16 20:54:47,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=9022, suffix heap size=142056,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=197 2023-04-16 20:54:47,515 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:47,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:47,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:47,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1681678487511, max timestamp=1681678487516Num uniques 1056; 2023-04-16 20:54:47,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=9430, suffix heap size=137792,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=206 2023-04-16 20:54:47,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1681678487511, max timestamp=1681678487516Num uniques 1056; 2023-04-16 20:54:47,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=9430, suffix heap size=137792,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=206 2023-04-16 20:54:47,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1681678487511, max timestamp=1681678487516Num uniques 1056; 2023-04-16 20:54:47,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=9430, suffix heap size=137792,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=206 2023-04-16 20:54:47,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:47,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:47,520 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:47,520 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,520 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:47,520 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:47,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,522 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,523 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,523 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:47,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:47,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:47,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:47,524 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,524 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:47,524 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:47,524 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:47,526 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e5a74ed[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:47,528 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:47,528 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:47,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:47,531 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:47,534 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:47,534 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:47,535 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:47,535 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:47,535 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:47,536 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,536 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,536 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,539 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681678487535, max timestamp=1681678487538Num uniques 1065; 2023-04-16 20:54:47,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=7550, suffix heap size=138728,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=165 2023-04-16 20:54:47,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681678487535, max timestamp=1681678487538Num uniques 1065; 2023-04-16 20:54:47,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=7550, suffix heap size=138728,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=165 2023-04-16 20:54:47,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681678487535, max timestamp=1681678487538Num uniques 1065; 2023-04-16 20:54:47,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=7550, suffix heap size=138728,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=165 2023-04-16 20:54:47,543 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,544 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:47,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,544 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:47,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,544 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:47,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=274, cellSize=12544, totalHeapSize=28736, min timestamp=1681678487549, max timestamp=1681678487559Num uniques 1024; 2023-04-16 20:54:47,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=12544, suffix heap size=134464,new segment heap size=28736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=274 2023-04-16 20:54:47,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=274, cellSize=12544, totalHeapSize=28736, min timestamp=1681678487549, max timestamp=1681678487559Num uniques 1024; 2023-04-16 20:54:47,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681678487549, max timestamp=1681678487559Num uniques 1024; 2023-04-16 20:54:47,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=12544, suffix heap size=134464,new segment heap size=28736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=274 2023-04-16 20:54:47,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-16 20:54:47,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1681678487573, max timestamp=1681678487582Num uniques 1133; 2023-04-16 20:54:47,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51888, new segment data size=10668, suffix heap size=145800,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1133, new segment cells count=233 2023-04-16 20:54:47,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1681678487573, max timestamp=1681678487582Num uniques 1083; 2023-04-16 20:54:47,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=10668, suffix heap size=140600,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=233 2023-04-16 20:54:47,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1681678487573, max timestamp=1681678487582Num uniques 1133; 2023-04-16 20:54:47,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51888, new segment data size=10668, suffix heap size=145800,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1133, new segment cells count=233 2023-04-16 20:54:47,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1681678487597, max timestamp=1681678487603Num uniques 1092; 2023-04-16 20:54:47,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=11082, suffix heap size=141536,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=242 2023-04-16 20:54:47,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11082, totalHeapSize=25408, min timestamp=1681678487597, max timestamp=1681678487603Num uniques 1092; 2023-04-16 20:54:47,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=11082, suffix heap size=141536,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=242 2023-04-16 20:54:47,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1681678487599, max timestamp=1681678487603Num uniques 1092; 2023-04-16 20:54:47,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=8792, suffix heap size=141536,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=192 2023-04-16 20:54:47,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:47,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1681678487620, max timestamp=1681678487625Num uniques 1101; 2023-04-16 20:54:47,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1681678487620, max timestamp=1681678487625Num uniques 1101; 2023-04-16 20:54:47,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=6915, suffix heap size=142472,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=151 2023-04-16 20:54:47,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=6915, suffix heap size=142472,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=151 2023-04-16 20:54:47,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678547624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1681678487620, max timestamp=1681678487625Num uniques 1051; 2023-04-16 20:54:47,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=6915, suffix heap size=137272,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=151 2023-04-16 20:54:47,644 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:47,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:47,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:47,648 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:47,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:47,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:47,649 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,649 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,651 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,652 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,652 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:47,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1681678487636, max timestamp=1681678487650Num uniques 1010; 2023-04-16 20:54:47,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=7320, suffix heap size=133008,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=160 2023-04-16 20:54:47,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1681678487636, max timestamp=1681678487650Num uniques 1010; 2023-04-16 20:54:47,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=7320, suffix heap size=133008,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=160 2023-04-16 20:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1681678487636, max timestamp=1681678487650Num uniques 1010; 2023-04-16 20:54:47,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=7320, suffix heap size=133008,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=160 2023-04-16 20:54:47,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:47,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:47,654 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,654 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:47,654 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:47,654 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,659 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,659 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:47,660 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:47,660 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6101045d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:47,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:47,662 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,662 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:47,664 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:47,666 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:47,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:47,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:47,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:47,666 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:47,668 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,668 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,668 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,668 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:47,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:47,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,672 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,672 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:47,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:47,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=219, cellSize=10024, totalHeapSize=23016, min timestamp=1681678487669, max timestamp=1681678487678Num uniques 1019; 2023-04-16 20:54:47,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=10024, suffix heap size=133944,new segment heap size=23016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=219 2023-04-16 20:54:47,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=219, cellSize=10024, totalHeapSize=23016, min timestamp=1681678487669, max timestamp=1681678487678Num uniques 1019; 2023-04-16 20:54:47,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=10024, suffix heap size=133944,new segment heap size=23016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=219 2023-04-16 20:54:47,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=219, cellSize=10024, totalHeapSize=23016, min timestamp=1681678487669, max timestamp=1681678487678Num uniques 1019; 2023-04-16 20:54:47,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=10024, suffix heap size=133944,new segment heap size=23016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=219 2023-04-16 20:54:47,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=178, cellSize=8148, totalHeapSize=18752, min timestamp=1681678487696, max timestamp=1681678487705Num uniques 1078; 2023-04-16 20:54:47,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=178, cellSize=8148, totalHeapSize=18752, min timestamp=1681678487696, max timestamp=1681678487705Num uniques 1078; 2023-04-16 20:54:47,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=178, cellSize=8148, totalHeapSize=18752, min timestamp=1681678487696, max timestamp=1681678487705Num uniques 1078; 2023-04-16 20:54:47,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49368, new segment data size=8148, suffix heap size=140080,new segment heap size=18752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1078, new segment cells count=178 2023-04-16 20:54:47,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49368, new segment data size=8148, suffix heap size=140080,new segment heap size=18752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1078, new segment cells count=178 2023-04-16 20:54:47,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49368, new segment data size=8148, suffix heap size=140080,new segment heap size=18752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1078, new segment cells count=178 2023-04-16 20:54:47,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1681678487721, max timestamp=1681678487730Num uniques 1037; 2023-04-16 20:54:47,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47492, new segment data size=10852, suffix heap size=135816,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1037, new segment cells count=237 2023-04-16 20:54:47,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8562, totalHeapSize=19688, min timestamp=1681678487721, max timestamp=1681678487730Num uniques 1037; 2023-04-16 20:54:47,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47492, new segment data size=8562, suffix heap size=135816,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1037, new segment cells count=187 2023-04-16 20:54:47,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8562, totalHeapSize=19688, min timestamp=1681678487721, max timestamp=1681678487730Num uniques 1037; 2023-04-16 20:54:47,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47492, new segment data size=8562, suffix heap size=135816,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1037, new segment cells count=187 2023-04-16 20:54:47,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681678487744, max timestamp=1681678487751Num uniques 1046; 2023-04-16 20:54:47,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=11266, suffix heap size=136752,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=246 2023-04-16 20:54:47,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681678487744, max timestamp=1681678487751Num uniques 1096; 2023-04-16 20:54:47,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=11266, suffix heap size=141952,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=246 2023-04-16 20:54:47,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681678487744, max timestamp=1681678487751Num uniques 1046; 2023-04-16 20:54:47,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=11266, suffix heap size=136752,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=246 2023-04-16 20:54:47,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/81f48d08ecdb4d6ca4637edfc76e378d 2023-04-16 20:54:47,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/1ad66a24220b4e4b895d7ce7f35a24f3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1ad66a24220b4e4b895d7ce7f35a24f3 2023-04-16 20:54:47,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1ad66a24220b4e4b895d7ce7f35a24f3, entries=150, sequenceid=572, filesize=11.9 K 2023-04-16 20:54:47,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/54bef9c99b7d432dbec41c413c0a7d2d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/54bef9c99b7d432dbec41c413c0a7d2d 2023-04-16 20:54:47,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/54bef9c99b7d432dbec41c413c0a7d2d, entries=150, sequenceid=572, filesize=11.9 K 2023-04-16 20:54:47,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/81f48d08ecdb4d6ca4637edfc76e378d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/81f48d08ecdb4d6ca4637edfc76e378d 2023-04-16 20:54:47,776 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:47,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:47,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:47,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/81f48d08ecdb4d6ca4637edfc76e378d, entries=150, sequenceid=572, filesize=11.9 K 2023-04-16 20:54:47,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:47,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:47,780 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:47,780 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,780 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:47,780 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:47,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=140.89 KB/144270 for 968ac8675f340fdac83d49b0dce63dda in 1314ms, sequenceid=572, compaction requested=false 2023-04-16 20:54:47,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:47,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:47,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=147.60 KB heapSize=416.67 KB 2023-04-16 20:54:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:47,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,784 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:47,784 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:54:47,784 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:47,784 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:54:47,784 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:47,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:47,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:47,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:54:47,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:47,787 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,788 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 466 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678547788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678547789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678547789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbb8b0b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:47,792 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:47,792 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,793 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:47,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:47,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:47,793 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,793 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:47,794 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=1353 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/29f22bd343f5444cb4f880532490f801 2023-04-16 20:54:47,794 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:47,798 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:47,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:47,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:47,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:47,799 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:47,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,802 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=1353 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/ae7430473601455b986a7f0816447975 2023-04-16 20:54:47,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:47,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,812 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,812 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:47,812 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:47,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,820 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=1353 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/45856d3e4f82427e878dede4d8542f52 2023-04-16 20:54:47,824 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/29f22bd343f5444cb4f880532490f801 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/29f22bd343f5444cb4f880532490f801 2023-04-16 20:54:47,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/29f22bd343f5444cb4f880532490f801, entries=250, sequenceid=1353, filesize=16.7 K 2023-04-16 20:54:47,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/ae7430473601455b986a7f0816447975 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/ae7430473601455b986a7f0816447975 2023-04-16 20:54:47,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/ae7430473601455b986a7f0816447975, entries=150, sequenceid=1353, filesize=11.9 K 2023-04-16 20:54:47,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/45856d3e4f82427e878dede4d8542f52 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/45856d3e4f82427e878dede4d8542f52 2023-04-16 20:54:47,834 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/45856d3e4f82427e878dede4d8542f52, entries=150, sequenceid=1353, filesize=11.9 K 2023-04-16 20:54:47,835 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~457.17 KB/468144, currentSize=20.13 KB/20610 for 968ac8675f340fdac83d49b0dce63dda in 51ms, sequenceid=1353, compaction requested=true 2023-04-16 20:54:47,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:47,835 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:47,835 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:47,835 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:47,835 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:47,835 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:47,836 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41984 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:47,836 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/A is initiating minor compaction (all files) 2023-04-16 20:54:47,836 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/A in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,836 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/2012de31779049308a986b85a462b1ab, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1ad66a24220b4e4b895d7ce7f35a24f3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/29f22bd343f5444cb4f880532490f801] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=41 K 2023-04-16 20:54:47,836 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:47,836 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/B is initiating minor compaction (all files) 2023-04-16 20:54:47,836 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2012de31779049308a986b85a462b1ab, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681678485357 2023-04-16 20:54:47,836 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/B in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,837 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3785df75078e40e09fbd087796fbe436, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/54bef9c99b7d432dbec41c413c0a7d2d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/ae7430473601455b986a7f0816447975] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=36.2 K 2023-04-16 20:54:47,837 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1ad66a24220b4e4b895d7ce7f35a24f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681678486017 2023-04-16 20:54:47,837 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3785df75078e40e09fbd087796fbe436, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681678485357 2023-04-16 20:54:47,837 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 29f22bd343f5444cb4f880532490f801, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=1353, earliestPutTs=1681678487765 2023-04-16 20:54:47,837 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 54bef9c99b7d432dbec41c413c0a7d2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681678486017 2023-04-16 20:54:47,838 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ae7430473601455b986a7f0816447975, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1353, earliestPutTs=1681678487778 2023-04-16 20:54:47,847 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#A#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:47,848 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#B#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:47,862 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/aafb27b35bb84fb8a984367d9d0c869a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/aafb27b35bb84fb8a984367d9d0c869a 2023-04-16 20:54:47,866 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/A of 968ac8675f340fdac83d49b0dce63dda into aafb27b35bb84fb8a984367d9d0c869a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:47,866 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:47,866 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/A, priority=13, startTime=1681678487835; duration=0sec 2023-04-16 20:54:47,866 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:47,867 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:47,867 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:47,867 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/C is initiating minor compaction (all files) 2023-04-16 20:54:47,868 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/C in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,868 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/8a51eb8ab7b94d16a5008f8127da78b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/81f48d08ecdb4d6ca4637edfc76e378d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/45856d3e4f82427e878dede4d8542f52] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=36.2 K 2023-04-16 20:54:47,868 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8a51eb8ab7b94d16a5008f8127da78b0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681678485357 2023-04-16 20:54:47,868 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 81f48d08ecdb4d6ca4637edfc76e378d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681678486017 2023-04-16 20:54:47,869 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 45856d3e4f82427e878dede4d8542f52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1353, earliestPutTs=1681678487778 2023-04-16 20:54:47,870 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/6d21bdd5b83a438eb4031c0a48c49127 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6d21bdd5b83a438eb4031c0a48c49127 2023-04-16 20:54:47,874 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/B of 968ac8675f340fdac83d49b0dce63dda into 6d21bdd5b83a438eb4031c0a48c49127(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:47,875 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:47,875 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/B, priority=13, startTime=1681678487835; duration=0sec 2023-04-16 20:54:47,875 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:47,876 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#C#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:47,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:47,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:54:47,897 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:47,897 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:47,897 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:47,897 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:47,897 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:47,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:47,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1366 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/c5dbb4bba54f466ba04d03bcdf9ad6e8 2023-04-16 20:54:47,912 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:47,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:47,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1366 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/9029c50297734ba4b382fdcada3bc0f7 2023-04-16 20:54:47,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:47,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:47,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:47,916 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:47,916 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,916 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:47,916 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:47,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,918 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678487909, max timestamp=1681678487917Num uniques 859; 2023-04-16 20:54:47,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:54:47,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678487909, max timestamp=1681678487917Num uniques 859; 2023-04-16 20:54:47,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:54:47,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678487909, max timestamp=1681678487917Num uniques 859; 2023-04-16 20:54:47,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:54:47,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:47,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:47,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:47,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:47,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:47,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:47,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:47,921 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,921 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:47,921 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:47,921 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:47,923 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59681edd[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:47,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:47,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,926 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,926 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:47,926 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:47,926 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:47,926 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,926 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:47,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:47,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:47,931 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:47,931 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:47,931 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:47,931 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:47,931 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:47,933 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:47,933 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,933 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:47,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,933 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,935 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,937 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,937 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:47,937 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:47,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:47,937 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:47,937 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:47,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:47,938 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:47,938 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:47,938 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:47,938 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,938 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:47,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:47,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678547935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,938 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:47,938 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:47,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:47,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678547938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:47,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:47,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678547938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,038 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:48,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:48,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:48,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:48,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:48,041 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:48,041 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:48,042 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:48,042 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:48,042 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,042 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678548039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,043 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:48,043 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:48,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:48,043 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:48,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:48,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:48,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:48,045 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:48,045 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:48,045 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:48,045 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:48,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:48,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678548043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:48,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678548043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,048 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:48,050 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:48,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3478e401[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:48,050 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:48,051 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:48,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:48,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:48,055 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:48,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:48,057 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:48,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:48,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:48,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:48,057 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:48,060 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,060 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,060 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:48,060 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,061 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:48,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:48,063 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:48,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,064 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,064 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,064 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,064 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:48,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:48,063 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:48,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:48,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:48,064 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:48,064 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:48,164 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:48,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:48,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:48,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:48,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:48,167 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:48,167 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:48,167 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:48,167 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:48,168 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,168 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,169 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:48,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:48,169 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,169 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,169 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:48,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:48,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:48,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:48,170 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:48,171 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:48,171 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:48,171 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:48,171 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:48,171 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,171 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:48,171 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:48,172 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,172 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,172 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,172 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:48,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db22d3f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:48,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:48,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,176 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:48,176 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,176 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:48,178 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:48,178 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:48,178 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,178 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:48,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:48,180 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:48,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:48,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:48,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:48,181 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:48,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,183 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,184 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:48,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:48,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:48,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:48,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:48,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:48,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:48,187 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,187 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:48,187 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,187 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:48,187 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,187 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:48,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678548244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678548246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678548247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,287 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/d67ce60d4fff4fdd98ffcdf05b4c790b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/d67ce60d4fff4fdd98ffcdf05b4c790b 2023-04-16 20:54:48,287 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:48,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:48,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:48,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:48,291 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/C of 968ac8675f340fdac83d49b0dce63dda into d67ce60d4fff4fdd98ffcdf05b4c790b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:48,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:48,291 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:48,291 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:48,291 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/C, priority=13, startTime=1681678487835; duration=0sec 2023-04-16 20:54:48,291 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:48,291 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:48,291 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:48,291 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:48,291 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,293 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,293 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:48,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:48,294 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,294 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:48,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:48,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:48,295 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:48,295 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:48,295 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:48,295 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:48,297 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:48,298 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e220651[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:48,299 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:48,300 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,300 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:48,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:48,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:48,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:48,303 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:48,303 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:48,303 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:48,303 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:48,303 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:48,306 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,306 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,306 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:48,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:48,307 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,308 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,310 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,310 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,310 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,310 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:48,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:48,310 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,310 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:48,310 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:48,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:48,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:48,310 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:48,310 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,310 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:48,310 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,310 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:48,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:48,325 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1366 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/ad3c543236dd4ecfbc90f01d07160813 2023-04-16 20:54:48,329 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/c5dbb4bba54f466ba04d03bcdf9ad6e8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c5dbb4bba54f466ba04d03bcdf9ad6e8 2023-04-16 20:54:48,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c5dbb4bba54f466ba04d03bcdf9ad6e8, entries=150, sequenceid=1366, filesize=11.9 K 2023-04-16 20:54:48,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/9029c50297734ba4b382fdcada3bc0f7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9029c50297734ba4b382fdcada3bc0f7 2023-04-16 20:54:48,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9029c50297734ba4b382fdcada3bc0f7, entries=150, sequenceid=1366, filesize=11.9 K 2023-04-16 20:54:48,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/ad3c543236dd4ecfbc90f01d07160813 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/ad3c543236dd4ecfbc90f01d07160813 2023-04-16 20:54:48,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/ad3c543236dd4ecfbc90f01d07160813, entries=150, sequenceid=1366, filesize=11.9 K 2023-04-16 20:54:48,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 968ac8675f340fdac83d49b0dce63dda in 445ms, sequenceid=1366, compaction requested=false 2023-04-16 20:54:48,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:48,410 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:48,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:48,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:48,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:48,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:48,413 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:48,414 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:48,414 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:48,414 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:48,414 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,414 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:48,415 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:48,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,416 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,416 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:48,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:48,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:48,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:48,417 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:48,417 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:48,417 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:48,421 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:48,421 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:48,421 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-16 20:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:48,422 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:48,431 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1402 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/ae29f475caa042439e52ec26e9e35c4f 2023-04-16 20:54:48,440 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1402 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/9d564851f026448b8cd203420b58e8d6 2023-04-16 20:54:48,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:48,545 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:48,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678548555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678548556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678548555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678548559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45518 deadline: 1681678548564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,566 DEBUG [Thread-1756] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:48,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 490 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678548659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678548660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678548660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678548662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,849 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1402 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/4fd97a79ebac4cb0ab66d87cc036bb9d 2023-04-16 20:54:48,853 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/ae29f475caa042439e52ec26e9e35c4f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ae29f475caa042439e52ec26e9e35c4f 2023-04-16 20:54:48,856 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ae29f475caa042439e52ec26e9e35c4f, entries=150, sequenceid=1402, filesize=11.9 K 2023-04-16 20:54:48,857 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/9d564851f026448b8cd203420b58e8d6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9d564851f026448b8cd203420b58e8d6 2023-04-16 20:54:48,861 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9d564851f026448b8cd203420b58e8d6, entries=150, sequenceid=1402, filesize=11.9 K 2023-04-16 20:54:48,861 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/4fd97a79ebac4cb0ab66d87cc036bb9d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4fd97a79ebac4cb0ab66d87cc036bb9d 2023-04-16 20:54:48,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 492 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678548863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678548864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678548864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:48,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678548866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,870 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4fd97a79ebac4cb0ab66d87cc036bb9d, entries=150, sequenceid=1402, filesize=11.9 K 2023-04-16 20:54:48,870 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 968ac8675f340fdac83d49b0dce63dda in 449ms, sequenceid=1402, compaction requested=true 2023-04-16 20:54:48,870 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:48,871 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:48,871 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:48,871 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:48,871 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,871 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:48,871 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:48,873 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37257bbe[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:48,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:48,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,878 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:48,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:48,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:48,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:48,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:48,882 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:48,882 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:48,882 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:48,882 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:48,882 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:48,886 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,886 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:48,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,886 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:48,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:48,887 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,891 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,891 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:48,891 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:48,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:48,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:48,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,892 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:48,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:48,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:48,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:48,892 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:48,892 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:48,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:48,992 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:48,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:48,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:48,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:48,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:48,996 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:48,996 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:48,996 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:48,996 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:48,996 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:48,996 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,997 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,997 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:48,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:48,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:48,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:48,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:49,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:49,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:49,000 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:49,000 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:49,001 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:54:49,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:49,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:49,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:49,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:49,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:49,001 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:49,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:49,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:49,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678549181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678489175, max timestamp=1681678489183Num uniques 859; 2023-04-16 20:54:49,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:54:49,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678489175, max timestamp=1681678489183Num uniques 859; 2023-04-16 20:54:49,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678489175, max timestamp=1681678489183Num uniques 859; 2023-04-16 20:54:49,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:54:49,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:54:49,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678549199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 504 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678549203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678549203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 451 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678549286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678549304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 506 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678549308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678549308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,419 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1413 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/e0bdf2c61e2949acb7e9be1be5dcd2fd 2023-04-16 20:54:49,428 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1413 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/3ef3288a951c4eaaa463ad41b3aadda2 2023-04-16 20:54:49,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678549489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678549508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 508 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678549510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678549511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678549793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678549811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 510 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678549813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:49,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 462 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678549814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,837 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1413 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/334a9138b04e466f843ff36bee535e8a 2023-04-16 20:54:49,842 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/e0bdf2c61e2949acb7e9be1be5dcd2fd as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/e0bdf2c61e2949acb7e9be1be5dcd2fd 2023-04-16 20:54:49,846 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/e0bdf2c61e2949acb7e9be1be5dcd2fd, entries=150, sequenceid=1413, filesize=11.9 K 2023-04-16 20:54:49,846 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/3ef3288a951c4eaaa463ad41b3aadda2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3ef3288a951c4eaaa463ad41b3aadda2 2023-04-16 20:54:49,850 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3ef3288a951c4eaaa463ad41b3aadda2, entries=150, sequenceid=1413, filesize=11.9 K 2023-04-16 20:54:49,851 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/334a9138b04e466f843ff36bee535e8a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/334a9138b04e466f843ff36bee535e8a 2023-04-16 20:54:49,854 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/334a9138b04e466f843ff36bee535e8a, entries=150, sequenceid=1413, filesize=11.9 K 2023-04-16 20:54:49,855 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 968ac8675f340fdac83d49b0dce63dda in 854ms, sequenceid=1413, compaction requested=true 2023-04-16 20:54:49,855 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:49,855 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:49,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:49,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:49,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:49,855 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:49,856 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:49,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:49,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@448ec9b7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:49,858 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:49,858 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:49,859 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:49,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:49,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:49,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:49,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:49,860 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,860 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:49,862 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:49,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:49,868 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:49,868 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:49,868 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:49,868 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:49,868 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:49,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:49,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:49,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,871 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,874 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,874 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:49,874 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:49,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:49,874 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:49,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:49,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:49,874 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:49,875 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:49,874 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:49,875 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:49,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:49,875 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,875 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:49,875 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:49,875 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,975 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:49,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:49,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:49,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:49,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:49,979 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:49,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:49,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:49,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:49,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:49,980 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,980 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:49,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:49,981 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:49,981 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:49,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:49,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:49,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:49,982 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:49,982 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:49,982 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-16 20:54:49,983 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:49,983 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:49,983 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:49,983 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:49,983 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:49,983 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:50,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:50,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:50,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:50,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 512 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678550316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:50,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:50,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678550317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:50,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:50,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678550319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:50,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:50,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678550320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:50,394 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1448 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/951ecaea85ff493e95b8ba74c26ce633 2023-04-16 20:54:50,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:50,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 466 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678550422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:50,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:50,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 468 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678550626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:50,803 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1448 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/4bde20bbc2b04474b174a9633837ba55 2023-04-16 20:54:50,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:50,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 470 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678550928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,222 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1448 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/59fd61dfe54f45c68a0681791756606c 2023-04-16 20:54:51,226 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/951ecaea85ff493e95b8ba74c26ce633 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/951ecaea85ff493e95b8ba74c26ce633 2023-04-16 20:54:51,229 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/951ecaea85ff493e95b8ba74c26ce633, entries=150, sequenceid=1448, filesize=11.9 K 2023-04-16 20:54:51,230 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/4bde20bbc2b04474b174a9633837ba55 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4bde20bbc2b04474b174a9633837ba55 2023-04-16 20:54:51,233 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4bde20bbc2b04474b174a9633837ba55, entries=150, sequenceid=1448, filesize=11.9 K 2023-04-16 20:54:51,234 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/59fd61dfe54f45c68a0681791756606c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/59fd61dfe54f45c68a0681791756606c 2023-04-16 20:54:51,237 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/59fd61dfe54f45c68a0681791756606c, entries=150, sequenceid=1448, filesize=11.9 K 2023-04-16 20:54:51,238 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 968ac8675f340fdac83d49b0dce63dda in 1256ms, sequenceid=1448, compaction requested=true 2023-04-16 20:54:51,238 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:51,238 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,238 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:51,238 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:51,239 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,239 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:51,239 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:51,241 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,241 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,241 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,241 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@819dfcc[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:51,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:51,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,244 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:51,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:51,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:51,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:51,249 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:51,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:51,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:51,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:51,249 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:51,250 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,250 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,250 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,253 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:51,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:51,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,259 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:51,259 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:51,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:51,323 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:54:51,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:51,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:51,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:51,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:51,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:51,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:51,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 520 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678551349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678491340, max timestamp=1681678491351Num uniques 859; 2023-04-16 20:54:51,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678491340, max timestamp=1681678491351Num uniques 859; 2023-04-16 20:54:51,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:54:51,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:54:51,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678491340, max timestamp=1681678491351Num uniques 859; 2023-04-16 20:54:51,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:54:51,359 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:51,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:51,362 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:51,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:51,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:51,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,363 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,364 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,364 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:51,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:51,365 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,365 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:51,365 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:51,365 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:51,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b75010[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:51,368 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:51,369 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,370 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:51,370 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,370 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:51,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:51,375 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:51,375 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:51,375 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:51,375 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:51,375 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:51,377 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,377 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,377 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,377 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,379 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:51,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,383 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:51,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,384 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,384 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,384 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:51,384 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:51,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678551387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678551387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678551431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 522 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678551454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,484 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:51,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:51,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:51,487 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:51,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:51,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:51,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,490 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,490 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,490 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:51,492 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:51,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:51,493 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:51,493 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,493 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:51,494 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:51,494 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,494 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:51,494 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:51,494 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,494 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:51,494 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:51,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678551491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678551491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,495 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,495 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,495 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,495 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:51,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a5325f3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:51,497 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:51,497 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,498 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:51,498 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,498 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:51,499 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:51,502 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:51,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:51,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:51,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:51,502 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:51,505 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,505 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,505 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,506 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,509 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,509 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,509 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,509 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:51,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:51,509 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,509 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,510 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,510 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,510 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,510 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:51,510 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:51,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,610 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:51,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:51,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:51,613 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:51,613 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,613 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:51,613 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:51,614 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,614 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,615 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:51,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:51,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:51,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:51,617 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,617 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:51,617 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:51,617 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:51,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,618 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,619 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,619 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:51,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:51,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1822927f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:51,622 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:51,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:51,623 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:51,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:51,626 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:51,626 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:51,626 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:51,626 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:51,626 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:51,629 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,629 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,629 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,630 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,631 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,633 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,633 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,633 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,633 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:51,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:51,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:51,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,633 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,634 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,634 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,634 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,634 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,634 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:51,634 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:51,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 524 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678551656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 483 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678551695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678551696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1459 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/0f7284d48aff4fc4b52826251f2a5cde 2023-04-16 20:54:51,734 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:51,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:51,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:51,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:51,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:51,738 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:51,738 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,738 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:51,738 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:51,740 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,740 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,745 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,745 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,745 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,745 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:51,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:51,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:51,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:51,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:51,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:51,748 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,748 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:51,748 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:51,748 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,751 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:51,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e83d908[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:51,755 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:51,755 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,757 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:51,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:51,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:51,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,758 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:51,764 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:51,764 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:51,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:51,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:51,765 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:51,767 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,767 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,767 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,767 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,769 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,771 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,771 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,771 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,771 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,771 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:51,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:51,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,771 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,771 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,772 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:51,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,772 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:51,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,872 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:51,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:51,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:51,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:51,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:51,875 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:51,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:51,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:51,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,879 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,880 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,880 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:51,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:51,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:51,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:51,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:51,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:51,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:51,881 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,881 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:51,881 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:51,881 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:51,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:51,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:51,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:51,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:51,889 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,889 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52382518[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:51,891 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:51,891 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,893 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,893 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:51,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,895 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:51,895 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:51,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:51,897 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:51,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:51,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:51,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:51,897 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:51,899 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,899 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:51,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,899 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,901 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,904 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:51,904 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:51,904 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,904 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:51,904 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:51,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:51,904 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:51,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:51,904 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:51,904 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,904 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:51,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:51,904 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:51,904 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:51,904 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:51,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:51,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:51,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 526 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678551961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678552000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 485 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678552000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,005 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:52,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:52,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:52,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:52,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:52,008 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:52,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:52,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:52,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,009 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:52,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:52,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:52,011 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,011 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:52,011 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:52,011 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:52,013 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61f7ee11[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:52,015 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:52,015 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,016 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:52,016 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,016 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:52,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:52,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:52,021 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:52,021 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:52,021 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:52,021 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:52,021 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:52,023 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,023 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,023 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,025 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:52,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,027 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,028 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:52,028 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:52,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,128 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:52,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:52,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:52,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:52,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:52,131 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:52,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:52,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:52,132 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,132 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,133 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,133 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,133 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,133 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:52,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:52,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:52,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,134 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:52,135 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,135 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:52,135 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:52,135 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:52,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,138 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,138 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,138 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dca293a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:52,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:52,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:52,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:52,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:52,145 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:52,145 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:52,145 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:52,145 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:52,146 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:52,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,147 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,149 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1459 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/9b245478c9f843f38c18247b887a9775 2023-04-16 20:54:52,152 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,152 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:52,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:52,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:52,152 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,153 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:52,152 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,153 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,153 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,153 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,153 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,153 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,153 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,153 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,253 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:52,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:52,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:52,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:52,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:52,256 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:52,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:52,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:52,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,257 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:52,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:52,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:52,259 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,259 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:52,259 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:52,259 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:52,263 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,263 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,263 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:52,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d41f612[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:52,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:52,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,267 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:52,267 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,267 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:52,267 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:52,267 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:52,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:52,270 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:52,270 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:52,270 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:52,270 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:52,270 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:52,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,272 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,273 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:52,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:52,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,275 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:52,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,276 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:52,275 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:52,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,376 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:52,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:52,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:52,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:52,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:52,379 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:52,379 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,379 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:52,379 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:52,379 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,379 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,382 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,382 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,383 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,383 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:52,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:52,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:52,384 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,384 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:52,384 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:52,384 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,386 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec21647[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:52,393 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:52,393 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,394 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:52,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:52,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:52,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:52,396 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:52,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:52,398 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:52,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:52,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:52,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:52,398 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:52,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,401 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,403 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,405 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:52,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:52,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,406 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:52,406 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:52,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:52,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 474 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678552435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:52,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 528 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678552466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,506 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:52,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:52,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:52,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:52,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678552506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:52,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 487 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678552507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:52,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:52,509 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:52,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:52,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:52,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,510 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,511 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,511 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,511 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:52,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:52,512 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,512 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:52,513 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:52,513 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:52,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:52,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:52,513 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:52,514 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,514 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:52,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601bf0e8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:52,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:52,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:52,519 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,519 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:52,521 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:52,523 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:52,523 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:52,523 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:52,523 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:52,523 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:52,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:52,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,525 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,526 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,529 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,529 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:52,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:52,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:52,529 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,529 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:52,529 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,529 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:52,529 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,529 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:52,529 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:52,529 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:52,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1459 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/2d525251db7a4546a413612ac570b09c 2023-04-16 20:54:52,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/0f7284d48aff4fc4b52826251f2a5cde as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0f7284d48aff4fc4b52826251f2a5cde 2023-04-16 20:54:52,573 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0f7284d48aff4fc4b52826251f2a5cde, entries=200, sequenceid=1459, filesize=14.3 K 2023-04-16 20:54:52,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/9b245478c9f843f38c18247b887a9775 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9b245478c9f843f38c18247b887a9775 2023-04-16 20:54:52,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9b245478c9f843f38c18247b887a9775, entries=150, sequenceid=1459, filesize=11.9 K 2023-04-16 20:54:52,577 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/2d525251db7a4546a413612ac570b09c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d525251db7a4546a413612ac570b09c 2023-04-16 20:54:52,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d525251db7a4546a413612ac570b09c, entries=150, sequenceid=1459, filesize=11.9 K 2023-04-16 20:54:52,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 968ac8675f340fdac83d49b0dce63dda in 1258ms, sequenceid=1459, compaction requested=true 2023-04-16 20:54:52,581 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:52,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:52,581 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:54:52,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:52,581 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:54:52,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:52,583 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76204 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:54:52,583 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/A is initiating minor compaction (all files) 2023-04-16 20:54:52,583 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/A in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,583 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/aafb27b35bb84fb8a984367d9d0c869a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c5dbb4bba54f466ba04d03bcdf9ad6e8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ae29f475caa042439e52ec26e9e35c4f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/e0bdf2c61e2949acb7e9be1be5dcd2fd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/951ecaea85ff493e95b8ba74c26ce633, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0f7284d48aff4fc4b52826251f2a5cde] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=74.4 K 2023-04-16 20:54:52,583 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:54:52,583 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/B is initiating minor compaction (all files) 2023-04-16 20:54:52,583 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/B in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,584 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6d21bdd5b83a438eb4031c0a48c49127, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9029c50297734ba4b382fdcada3bc0f7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9d564851f026448b8cd203420b58e8d6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3ef3288a951c4eaaa463ad41b3aadda2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4bde20bbc2b04474b174a9633837ba55, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9b245478c9f843f38c18247b887a9775] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=72.0 K 2023-04-16 20:54:52,584 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting aafb27b35bb84fb8a984367d9d0c869a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1353, earliestPutTs=1681678487778 2023-04-16 20:54:52,584 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6d21bdd5b83a438eb4031c0a48c49127, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1353, earliestPutTs=1681678487778 2023-04-16 20:54:52,584 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c5dbb4bba54f466ba04d03bcdf9ad6e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1366, earliestPutTs=1681678487788 2023-04-16 20:54:52,584 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9029c50297734ba4b382fdcada3bc0f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1366, earliestPutTs=1681678487788 2023-04-16 20:54:52,584 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ae29f475caa042439e52ec26e9e35c4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1402, earliestPutTs=1681678487934 2023-04-16 20:54:52,585 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9d564851f026448b8cd203420b58e8d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1402, earliestPutTs=1681678487934 2023-04-16 20:54:52,585 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e0bdf2c61e2949acb7e9be1be5dcd2fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1413, earliestPutTs=1681678488553 2023-04-16 20:54:52,585 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3ef3288a951c4eaaa463ad41b3aadda2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1413, earliestPutTs=1681678488553 2023-04-16 20:54:52,585 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 951ecaea85ff493e95b8ba74c26ce633, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1448, earliestPutTs=1681678489191 2023-04-16 20:54:52,585 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 4bde20bbc2b04474b174a9633837ba55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1448, earliestPutTs=1681678489191 2023-04-16 20:54:52,585 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 0f7284d48aff4fc4b52826251f2a5cde, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1459, earliestPutTs=1681678490308 2023-04-16 20:54:52,585 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9b245478c9f843f38c18247b887a9775, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1459, earliestPutTs=1681678490314 2023-04-16 20:54:52,615 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#A#compaction#303 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:52,616 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#B#compaction#304 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:52,629 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:52,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:52,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:52,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:52,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:52,637 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:52,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:52,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:52,639 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,639 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:52,641 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:52,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:52,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:52,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:52,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:52,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:52,647 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/c2f98e7fac2d46b5bc98f044c8e76234 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c2f98e7fac2d46b5bc98f044c8e76234 2023-04-16 20:54:52,652 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:52,652 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:52,652 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,653 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:52,653 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/A of 968ac8675f340fdac83d49b0dce63dda into c2f98e7fac2d46b5bc98f044c8e76234(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:52,653 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:52,653 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-16 20:54:52,653 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/A, priority=10, startTime=1681678492581; duration=0sec 2023-04-16 20:54:52,653 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:52,653 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:52,653 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:52,653 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:54:52,653 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:52,653 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:52,653 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:52,653 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:52,658 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:54:52,658 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/C is initiating minor compaction (all files) 2023-04-16 20:54:52,658 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/C in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:52,658 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/d67ce60d4fff4fdd98ffcdf05b4c790b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/ad3c543236dd4ecfbc90f01d07160813, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4fd97a79ebac4cb0ab66d87cc036bb9d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/334a9138b04e466f843ff36bee535e8a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/59fd61dfe54f45c68a0681791756606c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d525251db7a4546a413612ac570b09c] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=72.0 K 2023-04-16 20:54:52,659 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d67ce60d4fff4fdd98ffcdf05b4c790b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1353, earliestPutTs=1681678487778 2023-04-16 20:54:52,659 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ad3c543236dd4ecfbc90f01d07160813, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1366, earliestPutTs=1681678487788 2023-04-16 20:54:52,660 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 4fd97a79ebac4cb0ab66d87cc036bb9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1402, earliestPutTs=1681678487934 2023-04-16 20:54:52,660 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 334a9138b04e466f843ff36bee535e8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1413, earliestPutTs=1681678488553 2023-04-16 20:54:52,661 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 59fd61dfe54f45c68a0681791756606c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1448, earliestPutTs=1681678489191 2023-04-16 20:54:52,661 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2d525251db7a4546a413612ac570b09c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1459, earliestPutTs=1681678490314 2023-04-16 20:54:52,680 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1496 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/198df8e66934430f80b1923bc2bc01c5 2023-04-16 20:54:52,682 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#C#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:52,698 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/eba266c97c09476c9ae54142ce41929e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/eba266c97c09476c9ae54142ce41929e 2023-04-16 20:54:52,704 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/C of 968ac8675f340fdac83d49b0dce63dda into eba266c97c09476c9ae54142ce41929e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:52,704 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:52,704 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/C, priority=10, startTime=1681678492581; duration=0sec 2023-04-16 20:54:52,704 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:53,044 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/cb62e311987443219729f6beab1a701b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/cb62e311987443219729f6beab1a701b 2023-04-16 20:54:53,048 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/B of 968ac8675f340fdac83d49b0dce63dda into cb62e311987443219729f6beab1a701b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:53,049 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:53,049 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/B, priority=10, startTime=1681678492581; duration=0sec 2023-04-16 20:54:53,049 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:53,103 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1496 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/5f7b47f150474487a9415b66a3d2119d 2023-04-16 20:54:53,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:53,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:53,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:53,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 537 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678553492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,513 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1496 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/5e65e3ed6e814788a6d7f9583efc613d 2023-04-16 20:54:53,518 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/198df8e66934430f80b1923bc2bc01c5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/198df8e66934430f80b1923bc2bc01c5 2023-04-16 20:54:53,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:53,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678553517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:53,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 489 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678553520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,526 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/198df8e66934430f80b1923bc2bc01c5, entries=150, sequenceid=1496, filesize=11.9 K 2023-04-16 20:54:53,527 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/5f7b47f150474487a9415b66a3d2119d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5f7b47f150474487a9415b66a3d2119d 2023-04-16 20:54:53,530 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5f7b47f150474487a9415b66a3d2119d, entries=150, sequenceid=1496, filesize=11.9 K 2023-04-16 20:54:53,531 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/5e65e3ed6e814788a6d7f9583efc613d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/5e65e3ed6e814788a6d7f9583efc613d 2023-04-16 20:54:53,534 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/5e65e3ed6e814788a6d7f9583efc613d, entries=150, sequenceid=1496, filesize=11.9 K 2023-04-16 20:54:53,535 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 968ac8675f340fdac83d49b0dce63dda in 882ms, sequenceid=1496, compaction requested=false 2023-04-16 20:54:53,535 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:53,535 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,537 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca48628[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:53,539 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:53,539 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,540 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:53,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:53,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:53,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:53,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:53,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:53,547 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:53,547 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:53,547 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:53,547 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:53,547 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:53,548 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,549 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,549 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:53,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,550 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,552 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,552 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:53,552 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,552 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:53,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:53,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:53,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:53,552 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:53,552 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:53,552 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,553 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,553 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,553 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:53,553 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:53,553 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:53,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:53,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:53,597 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:53,598 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:53,598 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:53,598 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:53,598 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:53,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678493639, max timestamp=1681678493647Num uniques 859; 2023-04-16 20:54:53,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:53,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678493639, max timestamp=1681678493647Num uniques 859; 2023-04-16 20:54:53,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678493639, max timestamp=1681678493647Num uniques 859; 2023-04-16 20:54:53,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:53,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:53,653 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:53,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:53,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:53,656 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:53,656 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:53,656 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:53,656 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:53,656 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,657 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:53,658 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:53,659 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,659 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:53,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:53,661 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:53,661 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:53,661 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:53,661 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:53,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef71ac8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:53,665 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:53,665 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,666 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:53,666 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:53,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:53,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,667 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:53,668 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:53,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:53,671 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:53,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:53,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:53,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:53,671 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:53,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:53,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:53,673 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:53,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:53,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,677 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:53,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:53,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:53,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:53,677 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:53,677 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:53,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:53,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:53,678 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,678 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,678 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:53,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:53,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 573 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678553697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,778 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:53,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:53,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:53,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:53,781 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:53,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:53,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:53,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:53,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:53,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:53,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:53,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:53,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:53,784 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:53,784 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:53,784 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:53,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:53,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:53,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a95b7f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:53,789 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:53,789 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,790 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:53,793 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:53,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:53,795 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:53,795 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:53,795 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:53,795 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:53,795 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:53,797 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,797 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,797 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:53,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:53,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:53,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:53,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:53,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:53,801 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:53,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:53,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,801 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:53,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 575 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678553801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,802 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,802 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,802 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,902 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:53,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:53,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:53,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:53,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:53,906 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:53,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:53,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:53,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:53,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,906 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,907 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:53,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:53,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:53,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:53,909 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:53,909 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:53,910 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:53,910 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:53,912 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:53,914 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:53,914 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8a6d32[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:53,915 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:53,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:53,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:53,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:53,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:53,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:53,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:53,919 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:53,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:53,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:53,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:53,919 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:53,924 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,924 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,924 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:53,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:53,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:53,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,926 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:53,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,928 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:53,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:53,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:53,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:53,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:53,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:53,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:53,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,928 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:53,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:53,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:53,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:53,928 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:54,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:54,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 577 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678554003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1509 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/919089f3e93143e684a26ace99ca0448 2023-04-16 20:54:54,029 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:54,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:54,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:54,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:54,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:54,032 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:54,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:54,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:54,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,033 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:54,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:54,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:54,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:54,035 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,035 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:54,035 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:54,035 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:54,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,037 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,037 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,037 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:54,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1767332d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:54,038 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:54,038 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,040 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:54,041 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,041 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,042 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:54,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:54,044 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:54,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:54,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:54,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:54,044 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:54,045 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,045 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,046 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,046 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,047 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,056 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,056 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:54,056 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:54,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:54,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,056 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,056 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,057 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,057 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,057 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,057 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:54,057 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,057 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:54,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,157 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:54,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:54,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:54,160 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:54,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:54,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:54,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,163 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,163 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,164 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:54,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:54,165 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,165 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:54,165 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:54,165 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:54,166 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:54,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,167 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,167 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,167 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:54,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51004a53[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:54,169 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:54,169 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,170 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,171 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:54,171 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:54,171 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:54,171 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,171 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:54,172 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,172 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:54,172 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,172 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:54,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:54,174 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:54,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:54,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:54,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:54,174 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:54,177 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,177 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,177 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,177 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,178 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:54,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:54,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:54,182 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:54,182 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:54,282 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:54,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:54,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:54,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:54,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:54,285 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:54,285 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,285 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:54,286 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:54,286 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,286 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,287 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,287 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,287 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,287 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:54,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:54,289 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,289 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:54,289 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:54,289 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:54,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,291 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:54,293 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:54,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4be891af[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:54,293 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,295 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:54,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:54,295 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:54,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:54,296 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,296 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:54,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:54,299 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:54,299 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:54,299 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:54,299 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:54,299 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:54,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,302 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,303 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:54,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 579 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678554305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,308 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,308 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,308 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,308 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,308 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:54,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:54,308 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,308 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,308 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,308 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,308 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:54,308 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,308 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:54,408 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:54,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:54,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:54,412 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:54,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:54,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:54,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,413 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,413 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,414 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,414 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:54,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:54,415 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,415 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:54,415 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:54,415 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:54,415 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:54,416 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,417 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,417 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1509 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/6014638af9ff4a80875664b9798d92de 2023-04-16 20:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:54,418 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7737961d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:54,418 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,420 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,420 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:54,420 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:54,420 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:54,423 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:54,423 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,423 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,423 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:54,429 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:54,429 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:54,429 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:54,429 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:54,429 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:54,431 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,431 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,431 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,431 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,433 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,435 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,435 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,435 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,435 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:54,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:54,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:54,436 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,436 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,436 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,436 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:54,436 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:54,436 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,436 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,436 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:54,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45474 deadline: 1681678554444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,446 DEBUG [Thread-1752] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4126 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:54:54,536 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:54,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:54,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:54,539 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:54,539 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:54,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:54,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,540 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,541 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,541 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,541 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,541 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:54,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:54,542 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:54,542 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,542 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:54,543 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:54,543 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:54,543 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,543 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:54,543 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:54,543 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,543 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:54,543 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:54,543 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,543 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:54,543 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:54,547 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,547 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:54,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e72d8d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:54,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:54,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,550 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,550 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:54,550 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:54,552 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:54,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:54,554 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:54,554 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:54,554 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:54,554 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:54,554 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:54,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,558 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,561 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,561 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,561 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,561 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,561 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,561 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,561 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:54,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:54,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,561 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:54,561 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:54,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:54,661 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:54,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:54,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:54,664 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:54,664 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,664 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:54,665 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:54,665 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,665 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,666 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,667 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,667 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:54,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:54,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:54,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:54,668 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,668 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:54,668 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:54,668 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae6b630[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:54,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:54,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:54,675 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:54,678 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:54,678 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:54,678 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:54,678 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:54,678 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:54,679 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,679 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,680 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,683 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:54,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:54,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:54,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,684 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:54,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,684 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:54,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,784 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:54,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:54,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:54,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:54,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:54,787 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:54,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:54,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:54,788 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,788 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,795 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,795 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,795 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:54,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:54,797 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,797 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:54,797 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:54,797 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:54,798 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,799 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:54,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50c2b2a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:54,800 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:54,800 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:54,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,802 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,803 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:54,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:54,806 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:54,806 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:54,806 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:54,806 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:54,806 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:54,807 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,807 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,808 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:54,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 581 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678554808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,811 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:54,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:54,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,812 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:54,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:54,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,812 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:54,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,828 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1509 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/39dcd9e424dd4f1c93aae24150e15364 2023-04-16 20:54:54,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/919089f3e93143e684a26ace99ca0448 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/919089f3e93143e684a26ace99ca0448 2023-04-16 20:54:54,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/919089f3e93143e684a26ace99ca0448, entries=200, sequenceid=1509, filesize=14.3 K 2023-04-16 20:54:54,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/6014638af9ff4a80875664b9798d92de as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6014638af9ff4a80875664b9798d92de 2023-04-16 20:54:54,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6014638af9ff4a80875664b9798d92de, entries=150, sequenceid=1509, filesize=11.9 K 2023-04-16 20:54:54,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/39dcd9e424dd4f1c93aae24150e15364 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/39dcd9e424dd4f1c93aae24150e15364 2023-04-16 20:54:54,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/39dcd9e424dd4f1c93aae24150e15364, entries=150, sequenceid=1509, filesize=11.9 K 2023-04-16 20:54:54,845 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 968ac8675f340fdac83d49b0dce63dda in 1248ms, sequenceid=1509, compaction requested=true 2023-04-16 20:54:54,845 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:54,845 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:54,845 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:54,845 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:54,845 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:54,845 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:54,846 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39850 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:54,846 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/A is initiating minor compaction (all files) 2023-04-16 20:54:54,846 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/A in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,846 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c2f98e7fac2d46b5bc98f044c8e76234, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/198df8e66934430f80b1923bc2bc01c5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/919089f3e93143e684a26ace99ca0448] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=38.9 K 2023-04-16 20:54:54,846 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:54,846 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/B is initiating minor compaction (all files) 2023-04-16 20:54:54,846 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/B in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,847 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/cb62e311987443219729f6beab1a701b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5f7b47f150474487a9415b66a3d2119d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6014638af9ff4a80875664b9798d92de] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=36.5 K 2023-04-16 20:54:54,847 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c2f98e7fac2d46b5bc98f044c8e76234, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1459, earliestPutTs=1681678490314 2023-04-16 20:54:54,847 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting cb62e311987443219729f6beab1a701b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1459, earliestPutTs=1681678490314 2023-04-16 20:54:54,847 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 198df8e66934430f80b1923bc2bc01c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1496, earliestPutTs=1681678491381 2023-04-16 20:54:54,847 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5f7b47f150474487a9415b66a3d2119d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1496, earliestPutTs=1681678491381 2023-04-16 20:54:54,847 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 919089f3e93143e684a26ace99ca0448, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1509, earliestPutTs=1681678493482 2023-04-16 20:54:54,848 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6014638af9ff4a80875664b9798d92de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1509, earliestPutTs=1681678493482 2023-04-16 20:54:54,857 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#B#compaction#312 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:54,858 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#A#compaction#313 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:54,912 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:54,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:54,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:54,915 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:54,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:54,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:54,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:54,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:54,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:54,918 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,918 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:54,918 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 20:54:54,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:54,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:54,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:54,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:54,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:54,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:54,925 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1545 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/007398a7633647e086f5c52469b33789 2023-04-16 20:54:54,934 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1545 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/89c65a6488a94af79bdeb27ef8883278 2023-04-16 20:54:54,942 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1545 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/b9c924945ad44822b0d7012004f6859d 2023-04-16 20:54:54,945 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/007398a7633647e086f5c52469b33789 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/007398a7633647e086f5c52469b33789 2023-04-16 20:54:54,949 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/007398a7633647e086f5c52469b33789, entries=150, sequenceid=1545, filesize=11.9 K 2023-04-16 20:54:54,950 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/89c65a6488a94af79bdeb27ef8883278 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/89c65a6488a94af79bdeb27ef8883278 2023-04-16 20:54:54,953 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/89c65a6488a94af79bdeb27ef8883278, entries=150, sequenceid=1545, filesize=11.9 K 2023-04-16 20:54:54,953 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/b9c924945ad44822b0d7012004f6859d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/b9c924945ad44822b0d7012004f6859d 2023-04-16 20:54:54,956 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/b9c924945ad44822b0d7012004f6859d, entries=150, sequenceid=1545, filesize=11.9 K 2023-04-16 20:54:54,957 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 968ac8675f340fdac83d49b0dce63dda in 39ms, sequenceid=1545, compaction requested=true 2023-04-16 20:54:54,957 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:54,957 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:54,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:54,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:54,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:54,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:54,960 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec0d2e8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:54,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:54,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:54,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:54,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:54,965 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:54,968 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:54,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:54,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:54,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:54,968 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:54,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:54,970 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,971 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:54,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:54,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:54,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:54,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:54,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:54,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:54,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:54,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:54,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:54,975 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:54,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:54,976 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:54,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,076 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:55,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:55,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:55,079 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:55,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:55,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:55,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,081 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,081 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,081 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,081 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:55,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:55,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:55,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:55,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:55,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:55,083 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:55,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:55,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:55,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:55,084 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:55,086 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:55,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eab15f8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:55,086 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,087 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:55,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:55,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:55,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:55,093 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:55,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:55,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:55,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:55,093 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:55,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,096 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,099 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:55,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,099 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:55,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,099 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:55,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,199 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:55,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:55,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:55,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:55,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:55,203 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:55,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:55,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:55,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,205 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,205 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,205 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,205 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:55,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:55,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:55,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:55,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:55,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:55,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:55,207 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,207 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:55,207 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:55,207 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:55,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:55,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:55,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:55,211 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:55,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1143612b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:55,213 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:55,213 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,214 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:55,214 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:55,215 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:55,218 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:55,218 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:55,218 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:55,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:55,219 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:55,220 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,220 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,220 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,222 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:55,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:55,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:55,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,225 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:55,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,225 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:55,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,226 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,226 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,267 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/4dee25723b384b81a47efa5241c625c3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4dee25723b384b81a47efa5241c625c3 2023-04-16 20:54:55,268 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/6a83b56d991d40b69224a111e8ad97e0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6a83b56d991d40b69224a111e8ad97e0 2023-04-16 20:54:55,271 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/B of 968ac8675f340fdac83d49b0dce63dda into 4dee25723b384b81a47efa5241c625c3(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:55,271 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:55,271 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/B, priority=13, startTime=1681678494845; duration=0sec 2023-04-16 20:54:55,271 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:55,271 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:54:55,273 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:54:55,273 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/C is initiating minor compaction (all files) 2023-04-16 20:54:55,273 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/C in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,273 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/eba266c97c09476c9ae54142ce41929e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/5e65e3ed6e814788a6d7f9583efc613d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/39dcd9e424dd4f1c93aae24150e15364, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/b9c924945ad44822b0d7012004f6859d] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=48.4 K 2023-04-16 20:54:55,273 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting eba266c97c09476c9ae54142ce41929e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1459, earliestPutTs=1681678490314 2023-04-16 20:54:55,273 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5e65e3ed6e814788a6d7f9583efc613d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1496, earliestPutTs=1681678491381 2023-04-16 20:54:55,274 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 39dcd9e424dd4f1c93aae24150e15364, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1509, earliestPutTs=1681678493482 2023-04-16 20:54:55,274 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b9c924945ad44822b0d7012004f6859d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1545, earliestPutTs=1681678493670 2023-04-16 20:54:55,278 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/A of 968ac8675f340fdac83d49b0dce63dda into 6a83b56d991d40b69224a111e8ad97e0(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:55,278 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:55,278 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/A, priority=13, startTime=1681678494845; duration=0sec 2023-04-16 20:54:55,278 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:55,282 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#C#compaction#317 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:55,290 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/05a0d6a5f4e14eb48e365b9aa8c09ae5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/05a0d6a5f4e14eb48e365b9aa8c09ae5 2023-04-16 20:54:55,294 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/C of 968ac8675f340fdac83d49b0dce63dda into 05a0d6a5f4e14eb48e365b9aa8c09ae5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:55,294 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:55,294 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/C, priority=12, startTime=1681678494845; duration=0sec 2023-04-16 20:54:55,294 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:55,325 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:55,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:55,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:55,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:55,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:55,329 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:55,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:55,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:55,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,330 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,330 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,331 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,331 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:55,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:55,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:55,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:55,332 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,332 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:55,332 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:55,333 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:55,334 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,334 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:55,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75724d52[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:55,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:55,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,338 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:55,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:55,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:55,338 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,338 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:55,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,340 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:55,340 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,340 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:55,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:55,343 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:55,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:55,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:55,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:55,343 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:55,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,346 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,347 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,350 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:55,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:55,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,350 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,350 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:55,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:55,350 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:55,450 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:55,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:55,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:55,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:55,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:55,454 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:55,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:55,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:55,455 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,455 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,456 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,456 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:55,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:55,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:55,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:55,458 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,458 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:55,458 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:55,458 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:55,461 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d6d8686[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:55,463 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:55,463 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:55,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:55,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:55,467 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:55,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:55,472 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:55,472 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:55,472 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:55,472 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:55,472 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:55,474 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,474 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,474 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,479 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:55,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:55,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,480 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:55,480 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,480 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:55,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:55,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:54:55,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:55,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:55,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:55,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:55,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:55,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:55,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1562 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/0006bbe2f25647a3b58943e022600b92 2023-04-16 20:54:55,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678495557, max timestamp=1681678495562Num uniques 859; 2023-04-16 20:54:55,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:55,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678495557, max timestamp=1681678495562Num uniques 859; 2023-04-16 20:54:55,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:55,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678495557, max timestamp=1681678495562Num uniques 859; 2023-04-16 20:54:55,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:54:55,580 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:55,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:55,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:55,583 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:55,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:55,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:55,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,584 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,585 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,585 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:55,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:55,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:55,586 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,586 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:55,586 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:55,586 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:55,588 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:55,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9f5006[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:55,589 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:55,589 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,590 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:55,590 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,591 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,594 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:55,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:55,597 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:55,597 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:55,597 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:55,597 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:55,597 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:55,598 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,598 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,598 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,599 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:55,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:55,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:55,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,603 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:55,602 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,603 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:55,603 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,603 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,603 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,603 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:55,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678555615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:55,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 507 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678555619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,703 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:55,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:55,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:55,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:55,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:55,707 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:55,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:55,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:55,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,708 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:55,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:55,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:55,710 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,710 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:55,710 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:55,710 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:55,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:55,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41df3a67[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:55,714 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:55,714 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,715 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:55,715 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,715 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:55,717 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:55,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:55,719 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:55,719 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:55,719 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:55,719 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:55,719 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:55,720 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,720 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,720 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,721 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:55,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678555720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 509 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678555722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,729 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:55,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:55,729 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,729 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:55,729 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,729 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,729 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,729 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:55,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:55,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:55,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 583 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45516 deadline: 1681678555820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,829 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:55,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:55,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:55,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:55,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:55,832 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:55,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:55,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:55,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,834 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,834 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,835 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,835 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:55,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:55,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:55,836 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,836 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:55,836 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:55,836 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:55,838 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,838 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,838 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:55,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2620e7d3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:55,840 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:55,840 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,841 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:55,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:55,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:55,841 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,841 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:55,843 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:55,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:55,845 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:55,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:55,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:55,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:55,846 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:55,847 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,847 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,847 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,848 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,849 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,851 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,851 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,851 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,851 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,851 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:55,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:55,851 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:55,851 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,851 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,851 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,851 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:55,851 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:55,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,851 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:55,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678555923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:55,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 511 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678555926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,951 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:55,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:55,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:55,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:55,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:55,955 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:55,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:55,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:55,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,956 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:55,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:55,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:55,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:55,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:55,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:55,958 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,958 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:55,958 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:55,958 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:55,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:55,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:55,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:55,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:55,961 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:55,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77280c27[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:55,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:55,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:55,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:55,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:55,965 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:55,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:55,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:55,968 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:55,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:55,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:55,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:55,968 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:55,969 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1562 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/b7316eb1135d49da91c050173585fd24 2023-04-16 20:54:55,973 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,973 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:55,973 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:55,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:55,974 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:55,977 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,977 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:55,978 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:55,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:55,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:55,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:55,977 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:55,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:55,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:55,978 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:55,978 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:55,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:55,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:55,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:55,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:56,078 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:56,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:56,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:56,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:56,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:56,081 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:56,081 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:56,081 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:56,081 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:56,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,083 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:56,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:56,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:56,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:56,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:56,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:56,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:56,085 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:56,085 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:56,085 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:56,085 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:56,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:56,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:56,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:56,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:56,086 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:56,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:56,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:56,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46067da4[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:56,088 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:56,088 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:56,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:56,091 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,091 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:56,091 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:56,091 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:56,094 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:56,094 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:56,094 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:56,094 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:56,094 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:56,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:56,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:56,096 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:56,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,099 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:56,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:56,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:56,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:56,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:56,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:56,099 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:56,099 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:56,199 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:56,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:56,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:56,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:56,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:56,203 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:56,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:56,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:56,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:56,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,207 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:56,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:56,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:56,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:56,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:56,209 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:56,209 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:56,209 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:56,209 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:56,210 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e327b01[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:56,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:56,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:56,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:56,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:56,213 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,213 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:56,215 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:56,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:56,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:56,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:56,217 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:56,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:56,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:56,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:56,217 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:56,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:56,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:56,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,221 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:56,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:56,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:56,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,223 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:56,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:56,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:56,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:56,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:56,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,223 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:56,223 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45484 deadline: 1681678556226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:54:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 513 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45498 deadline: 1681678556227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,323 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:56,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:56,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:56,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:56,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:56,326 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:56,326 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:56,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:56,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:56,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,327 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,328 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:56,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:56,328 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:56,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:56,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:56,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:56,330 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:56,330 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:56,330 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:56,330 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:56,331 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,331 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:56,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f0d156b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:56,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:56,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,335 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:56,336 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:56,336 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:56,335 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:56,337 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:56,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:56,339 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:56,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:56,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:56,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:56,339 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:56,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:56,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:56,341 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,342 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,347 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,348 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,348 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:56,348 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:56,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:56,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:56,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:56,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:56,348 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:56,348 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:56,348 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:56,348 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,348 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,348 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,348 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:56,348 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:56,381 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1562 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/924d41709fd344769ff0a0b49af814e5 2023-04-16 20:54:56,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/0006bbe2f25647a3b58943e022600b92 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0006bbe2f25647a3b58943e022600b92 2023-04-16 20:54:56,388 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0006bbe2f25647a3b58943e022600b92, entries=150, sequenceid=1562, filesize=11.9 K 2023-04-16 20:54:56,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/b7316eb1135d49da91c050173585fd24 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b7316eb1135d49da91c050173585fd24 2023-04-16 20:54:56,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b7316eb1135d49da91c050173585fd24, entries=100, sequenceid=1562, filesize=9.5 K 2023-04-16 20:54:56,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/924d41709fd344769ff0a0b49af814e5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/924d41709fd344769ff0a0b49af814e5 2023-04-16 20:54:56,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/924d41709fd344769ff0a0b49af814e5, entries=100, sequenceid=1562, filesize=9.5 K 2023-04-16 20:54:56,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 968ac8675f340fdac83d49b0dce63dda in 860ms, sequenceid=1562, compaction requested=true 2023-04-16 20:54:56,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:56,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:56,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:56,398 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:56,398 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:54:56,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:56,399 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37508 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:56,399 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:54:56,399 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/A is initiating minor compaction (all files) 2023-04-16 20:54:56,399 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 968ac8675f340fdac83d49b0dce63dda/B is initiating minor compaction (all files) 2023-04-16 20:54:56,399 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/A in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:56,399 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 968ac8675f340fdac83d49b0dce63dda/B in TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:56,399 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6a83b56d991d40b69224a111e8ad97e0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/007398a7633647e086f5c52469b33789, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0006bbe2f25647a3b58943e022600b92] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=36.6 K 2023-04-16 20:54:56,399 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4dee25723b384b81a47efa5241c625c3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/89c65a6488a94af79bdeb27ef8883278, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b7316eb1135d49da91c050173585fd24] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp, totalSize=34.2 K 2023-04-16 20:54:56,400 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6a83b56d991d40b69224a111e8ad97e0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=1509, earliestPutTs=1681678493482 2023-04-16 20:54:56,400 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 4dee25723b384b81a47efa5241c625c3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=1509, earliestPutTs=1681678493482 2023-04-16 20:54:56,400 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 007398a7633647e086f5c52469b33789, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1545, earliestPutTs=1681678493670 2023-04-16 20:54:56,400 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 89c65a6488a94af79bdeb27ef8883278, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1545, earliestPutTs=1681678493670 2023-04-16 20:54:56,400 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 0006bbe2f25647a3b58943e022600b92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1562, earliestPutTs=1681678495535 2023-04-16 20:54:56,401 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b7316eb1135d49da91c050173585fd24, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1562, earliestPutTs=1681678495536 2023-04-16 20:54:56,407 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#B#compaction#321 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:56,413 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 968ac8675f340fdac83d49b0dce63dda#A#compaction#322 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:54:56,424 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/a9864f8968df4a41a83aa386307257e2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/a9864f8968df4a41a83aa386307257e2 2023-04-16 20:54:56,428 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/A of 968ac8675f340fdac83d49b0dce63dda into a9864f8968df4a41a83aa386307257e2(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:56,428 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:56,428 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/A, priority=13, startTime=1681678496398; duration=0sec 2023-04-16 20:54:56,428 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:54:56,428 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-16 20:54:56,429 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-16 20:54:56,429 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-16 20:54:56,429 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. because compaction request was cancelled 2023-04-16 20:54:56,448 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:56,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:56,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:54:56,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:54:56,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:54:56,451 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:54:56,451 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:56,451 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:54:56,451 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:54:56,452 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,452 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,453 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:56,453 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:56,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:56,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:54:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:54:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:56,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:54:56,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:54:56,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:54:56,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:54:56,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:54:56,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:54:56,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:54:56,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:54:56,455 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:56,455 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. started... 2023-04-16 20:54:56,455 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 20:54:56,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:56,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:56,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:56,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:56,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:56,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:54:56,521 DEBUG [Thread-1769] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de064cb to 127.0.0.1:54578 2023-04-16 20:54:56,521 DEBUG [Thread-1769] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:56,522 DEBUG [Thread-1767] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:54578 2023-04-16 20:54:56,522 DEBUG [Thread-1767] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:56,522 DEBUG [Thread-1771] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54a39be8 to 127.0.0.1:54578 2023-04-16 20:54:56,522 DEBUG [Thread-1771] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:56,523 DEBUG [Thread-1765] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:54578 2023-04-16 20:54:56,523 DEBUG [Thread-1765] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:56,524 DEBUG [Thread-1763] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:54578 2023-04-16 20:54:56,524 DEBUG [Thread-1763] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:56,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. as already flushing 2023-04-16 20:54:56,730 DEBUG [Thread-1754] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:54578 2023-04-16 20:54:56,730 DEBUG [Thread-1754] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:56,733 DEBUG [Thread-1760] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08cd7cb0 to 127.0.0.1:54578 2023-04-16 20:54:56,733 DEBUG [Thread-1760] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:56,826 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/d0ed9047cba0401d9c07d9cc01efe4b3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/d0ed9047cba0401d9c07d9cc01efe4b3 2023-04-16 20:54:56,829 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 968ac8675f340fdac83d49b0dce63dda/B of 968ac8675f340fdac83d49b0dce63dda into d0ed9047cba0401d9c07d9cc01efe4b3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:54:56,829 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:56,829 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda., storeName=968ac8675f340fdac83d49b0dce63dda/B, priority=13, startTime=1681678496398; duration=0sec 2023-04-16 20:54:56,829 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:54:56,863 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1600 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/4419fecf3f2d4a2fb0827d6e38c8ccad 2023-04-16 20:54:57,272 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1600 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/0a49b9c7822342689fb344ad16b9af68 2023-04-16 20:54:57,682 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1600 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/726a31a98a6344f18c7c079efa36b13b 2023-04-16 20:54:57,686 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/4419fecf3f2d4a2fb0827d6e38c8ccad as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4419fecf3f2d4a2fb0827d6e38c8ccad 2023-04-16 20:54:57,689 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4419fecf3f2d4a2fb0827d6e38c8ccad, entries=150, sequenceid=1600, filesize=11.9 K 2023-04-16 20:54:57,690 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/0a49b9c7822342689fb344ad16b9af68 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/0a49b9c7822342689fb344ad16b9af68 2023-04-16 20:54:57,693 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/0a49b9c7822342689fb344ad16b9af68, entries=150, sequenceid=1600, filesize=11.9 K 2023-04-16 20:54:57,694 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/726a31a98a6344f18c7c079efa36b13b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/726a31a98a6344f18c7c079efa36b13b 2023-04-16 20:54:57,697 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/726a31a98a6344f18c7c079efa36b13b, entries=150, sequenceid=1600, filesize=11.9 K 2023-04-16 20:54:57,698 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=13.42 KB/13740 for 968ac8675f340fdac83d49b0dce63dda in 1243ms, sequenceid=1600, compaction requested=true 2023-04-16 20:54:57,698 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:54:57,698 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:54:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:57,700 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68736399[Count = 0] remaining members to acquire global barrier 2023-04-16 20:54:57,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:54:57,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:57,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:54:57,703 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:54:57,703 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:54:57,703 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,703 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:54:57,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:54:57,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:57,704 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:54:57,706 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:54:57,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:54:57,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:54:57,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:54:57,706 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:54:57,709 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:57,709 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:54:57,709 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:57,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:54:57,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,710 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:57,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:54:57,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:57,712 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:54:57,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:54:57,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:54:57,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:54:57,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:54:57,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:57,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:54:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:54:57,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:54:57,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:54:57,712 DEBUG [Thread-1762] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:54:57,713 DEBUG [Thread-1762] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:54:57,713 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:54:57,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:54:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:54:57,813 DEBUG [Thread-1762] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:54:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:54:57,832 DEBUG [Thread-1758] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:54578 2023-04-16 20:54:57,832 DEBUG [Thread-1758] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:58,449 DEBUG [Thread-1752] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:54578 2023-04-16 20:54:58,449 DEBUG [Thread-1752] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:58,577 DEBUG [Thread-1756] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:54578 2023-04-16 20:54:58,577 DEBUG [Thread-1756] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 375 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 153 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 478 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 404 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 5342 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 5251 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 5110 2023-04-16 20:54:58,577 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 5339 2023-04-16 20:54:58,578 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 5287 2023-04-16 20:54:58,578 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 20:54:58,578 INFO [Listener at localhost/45179] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 20:54:58,578 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:54578 2023-04-16 20:54:58,578 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:54:58,578 INFO [Listener at localhost/45179] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 20:54:58,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 20:54:58,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 20:54:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 20:54:58,582 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678498582"}]},"ts":"1681678498582"} 2023-04-16 20:54:58,582 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 20:54:58,584 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 20:54:58,584 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=968ac8675f340fdac83d49b0dce63dda, UNASSIGN}] 2023-04-16 20:54:58,585 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=968ac8675f340fdac83d49b0dce63dda, UNASSIGN 2023-04-16 20:54:58,586 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=968ac8675f340fdac83d49b0dce63dda, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:54:58,586 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678498586"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678498586"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678498586"}]},"ts":"1681678498586"} 2023-04-16 20:54:58,587 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:54:58,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 20:54:58,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:54:58,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 968ac8675f340fdac83d49b0dce63dda, disabling compactions & flushes 2023-04-16 20:54:58,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:58,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:58,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. after waiting 0 ms 2023-04-16 20:54:58,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:54:58,739 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 968ac8675f340fdac83d49b0dce63dda 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-16 20:54:58,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=A 2023-04-16 20:54:58,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:58,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=B 2023-04-16 20:54:58,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:58,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 968ac8675f340fdac83d49b0dce63dda, store=C 2023-04-16 20:54:58,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:54:58,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 20:54:59,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1609 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/eccac1d3abff40e5ace2a06a9e18d551 2023-04-16 20:54:59,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 20:54:59,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1609 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/284f71bc4af24681ab39b37b725b020a 2023-04-16 20:54:59,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 20:54:59,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1609 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/224e296496a24e249df86d70edf5f068 2023-04-16 20:54:59,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/A/eccac1d3abff40e5ace2a06a9e18d551 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/eccac1d3abff40e5ace2a06a9e18d551 2023-04-16 20:54:59,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/eccac1d3abff40e5ace2a06a9e18d551, entries=100, sequenceid=1609, filesize=9.5 K 2023-04-16 20:54:59,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/B/284f71bc4af24681ab39b37b725b020a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/284f71bc4af24681ab39b37b725b020a 2023-04-16 20:54:59,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/284f71bc4af24681ab39b37b725b020a, entries=100, sequenceid=1609, filesize=9.5 K 2023-04-16 20:54:59,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/.tmp/C/224e296496a24e249df86d70edf5f068 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/224e296496a24e249df86d70edf5f068 2023-04-16 20:54:59,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/224e296496a24e249df86d70edf5f068, entries=100, sequenceid=1609, filesize=9.5 K 2023-04-16 20:54:59,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 968ac8675f340fdac83d49b0dce63dda in 1238ms, sequenceid=1609, compaction requested=true 2023-04-16 20:54:59,996 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/04004f2546c847199d44f1d346e6aa19, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/163b3acdd2ba4cf7ad3da40d71863d50, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6421c4d15d5241969c29ba9d4130352b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ea01a599220143258ff13ffa1c040cce, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/870d35d40c434659978be1d36dce4caf, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/9fef556984b044a4959ce078aac3a64c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ee8f30cebf434b6dbaccb16d9cb7cd30, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/75843512ddd642109ab02585bae0f47c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/8c0e7d2086f7457b843aafdc58028de8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c0e0b84d65c544f5ad233db429da00b5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/10b35ba7e0904abfbcc9ae5e05e8d5e0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/a64c15e1bf6346cba86b8f720ec3eb9f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/7736b31809fa4038ace35bc345a975ea, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4526cfecd30d4a629b15023e9032f622, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/17eedce6a5264fbd88ca8a5fab8bfabe, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1eca3a92663b47f6bceaba2078cb86a0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/2012de31779049308a986b85a462b1ab, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1ad66a24220b4e4b895d7ce7f35a24f3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/29f22bd343f5444cb4f880532490f801, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/aafb27b35bb84fb8a984367d9d0c869a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c5dbb4bba54f466ba04d03bcdf9ad6e8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ae29f475caa042439e52ec26e9e35c4f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/e0bdf2c61e2949acb7e9be1be5dcd2fd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/951ecaea85ff493e95b8ba74c26ce633, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0f7284d48aff4fc4b52826251f2a5cde, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c2f98e7fac2d46b5bc98f044c8e76234, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/198df8e66934430f80b1923bc2bc01c5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/919089f3e93143e684a26ace99ca0448, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6a83b56d991d40b69224a111e8ad97e0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/007398a7633647e086f5c52469b33789, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0006bbe2f25647a3b58943e022600b92] to archive 2023-04-16 20:54:59,997 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:54:59,998 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/04004f2546c847199d44f1d346e6aa19 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/04004f2546c847199d44f1d346e6aa19 2023-04-16 20:54:59,999 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/163b3acdd2ba4cf7ad3da40d71863d50 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/163b3acdd2ba4cf7ad3da40d71863d50 2023-04-16 20:55:00,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6421c4d15d5241969c29ba9d4130352b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6421c4d15d5241969c29ba9d4130352b 2023-04-16 20:55:00,001 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ea01a599220143258ff13ffa1c040cce to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ea01a599220143258ff13ffa1c040cce 2023-04-16 20:55:00,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/870d35d40c434659978be1d36dce4caf to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/870d35d40c434659978be1d36dce4caf 2023-04-16 20:55:00,003 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/9fef556984b044a4959ce078aac3a64c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/9fef556984b044a4959ce078aac3a64c 2023-04-16 20:55:00,004 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ee8f30cebf434b6dbaccb16d9cb7cd30 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ee8f30cebf434b6dbaccb16d9cb7cd30 2023-04-16 20:55:00,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/75843512ddd642109ab02585bae0f47c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/75843512ddd642109ab02585bae0f47c 2023-04-16 20:55:00,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/8c0e7d2086f7457b843aafdc58028de8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/8c0e7d2086f7457b843aafdc58028de8 2023-04-16 20:55:00,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c0e0b84d65c544f5ad233db429da00b5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c0e0b84d65c544f5ad233db429da00b5 2023-04-16 20:55:00,007 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/10b35ba7e0904abfbcc9ae5e05e8d5e0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/10b35ba7e0904abfbcc9ae5e05e8d5e0 2023-04-16 20:55:00,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/a64c15e1bf6346cba86b8f720ec3eb9f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/a64c15e1bf6346cba86b8f720ec3eb9f 2023-04-16 20:55:00,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/7736b31809fa4038ace35bc345a975ea to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/7736b31809fa4038ace35bc345a975ea 2023-04-16 20:55:00,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4526cfecd30d4a629b15023e9032f622 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4526cfecd30d4a629b15023e9032f622 2023-04-16 20:55:00,010 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/17eedce6a5264fbd88ca8a5fab8bfabe to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/17eedce6a5264fbd88ca8a5fab8bfabe 2023-04-16 20:55:00,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1eca3a92663b47f6bceaba2078cb86a0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1eca3a92663b47f6bceaba2078cb86a0 2023-04-16 20:55:00,012 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/2012de31779049308a986b85a462b1ab to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/2012de31779049308a986b85a462b1ab 2023-04-16 20:55:00,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1ad66a24220b4e4b895d7ce7f35a24f3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/1ad66a24220b4e4b895d7ce7f35a24f3 2023-04-16 20:55:00,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/29f22bd343f5444cb4f880532490f801 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/29f22bd343f5444cb4f880532490f801 2023-04-16 20:55:00,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/aafb27b35bb84fb8a984367d9d0c869a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/aafb27b35bb84fb8a984367d9d0c869a 2023-04-16 20:55:00,015 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c5dbb4bba54f466ba04d03bcdf9ad6e8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c5dbb4bba54f466ba04d03bcdf9ad6e8 2023-04-16 20:55:00,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ae29f475caa042439e52ec26e9e35c4f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/ae29f475caa042439e52ec26e9e35c4f 2023-04-16 20:55:00,017 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/e0bdf2c61e2949acb7e9be1be5dcd2fd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/e0bdf2c61e2949acb7e9be1be5dcd2fd 2023-04-16 20:55:00,017 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/951ecaea85ff493e95b8ba74c26ce633 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/951ecaea85ff493e95b8ba74c26ce633 2023-04-16 20:55:00,018 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0f7284d48aff4fc4b52826251f2a5cde to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0f7284d48aff4fc4b52826251f2a5cde 2023-04-16 20:55:00,019 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c2f98e7fac2d46b5bc98f044c8e76234 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/c2f98e7fac2d46b5bc98f044c8e76234 2023-04-16 20:55:00,020 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/198df8e66934430f80b1923bc2bc01c5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/198df8e66934430f80b1923bc2bc01c5 2023-04-16 20:55:00,021 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/919089f3e93143e684a26ace99ca0448 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/919089f3e93143e684a26ace99ca0448 2023-04-16 20:55:00,021 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6a83b56d991d40b69224a111e8ad97e0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/6a83b56d991d40b69224a111e8ad97e0 2023-04-16 20:55:00,022 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/007398a7633647e086f5c52469b33789 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/007398a7633647e086f5c52469b33789 2023-04-16 20:55:00,023 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0006bbe2f25647a3b58943e022600b92 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/0006bbe2f25647a3b58943e022600b92 2023-04-16 20:55:00,025 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/79aee9c23caf4a93a049657a363ccedc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c79503620c0445048d46ed74673939fe, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c133ff63ee11401290c6f1158b98dbdd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/76329a41a2724fb2adddedc7e1e15d28, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/d8d0e9bf352c4997a4957806a273ef4a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/71e9a6ec7a5741588c4078453b2ee869, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9fc6e204e9454cd193104eeaed892e5f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c20c47e7d4b54609a032aef576c1a672, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/00c10d3205da4f1f9b66868da1f3fcfc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/dd9a4f384ba64717b2fefc6df1a2119b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/8b11eda4d100431a906bab2bf40dacad, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5957dacf663f4fcc9d7945388510381b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c59a9d0fe26042e69791d4400cfbccf9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/15c3cb1ae5d94446b8569671c68c6991, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b70a2bdfb77b426d89a06ef273da1391, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3785df75078e40e09fbd087796fbe436, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/22143937bcfa466e80bcde0629def12c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/54bef9c99b7d432dbec41c413c0a7d2d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6d21bdd5b83a438eb4031c0a48c49127, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/ae7430473601455b986a7f0816447975, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9029c50297734ba4b382fdcada3bc0f7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9d564851f026448b8cd203420b58e8d6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3ef3288a951c4eaaa463ad41b3aadda2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4bde20bbc2b04474b174a9633837ba55, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/cb62e311987443219729f6beab1a701b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9b245478c9f843f38c18247b887a9775, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5f7b47f150474487a9415b66a3d2119d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4dee25723b384b81a47efa5241c625c3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6014638af9ff4a80875664b9798d92de, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/89c65a6488a94af79bdeb27ef8883278, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b7316eb1135d49da91c050173585fd24] to archive 2023-04-16 20:55:00,026 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:00,027 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/79aee9c23caf4a93a049657a363ccedc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/79aee9c23caf4a93a049657a363ccedc 2023-04-16 20:55:00,028 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c79503620c0445048d46ed74673939fe to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c79503620c0445048d46ed74673939fe 2023-04-16 20:55:00,029 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c133ff63ee11401290c6f1158b98dbdd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c133ff63ee11401290c6f1158b98dbdd 2023-04-16 20:55:00,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/76329a41a2724fb2adddedc7e1e15d28 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/76329a41a2724fb2adddedc7e1e15d28 2023-04-16 20:55:00,031 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/d8d0e9bf352c4997a4957806a273ef4a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/d8d0e9bf352c4997a4957806a273ef4a 2023-04-16 20:55:00,032 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/71e9a6ec7a5741588c4078453b2ee869 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/71e9a6ec7a5741588c4078453b2ee869 2023-04-16 20:55:00,033 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9fc6e204e9454cd193104eeaed892e5f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9fc6e204e9454cd193104eeaed892e5f 2023-04-16 20:55:00,033 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c20c47e7d4b54609a032aef576c1a672 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c20c47e7d4b54609a032aef576c1a672 2023-04-16 20:55:00,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/00c10d3205da4f1f9b66868da1f3fcfc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/00c10d3205da4f1f9b66868da1f3fcfc 2023-04-16 20:55:00,035 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/dd9a4f384ba64717b2fefc6df1a2119b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/dd9a4f384ba64717b2fefc6df1a2119b 2023-04-16 20:55:00,036 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/8b11eda4d100431a906bab2bf40dacad to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/8b11eda4d100431a906bab2bf40dacad 2023-04-16 20:55:00,036 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5957dacf663f4fcc9d7945388510381b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5957dacf663f4fcc9d7945388510381b 2023-04-16 20:55:00,037 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c59a9d0fe26042e69791d4400cfbccf9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/c59a9d0fe26042e69791d4400cfbccf9 2023-04-16 20:55:00,038 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/15c3cb1ae5d94446b8569671c68c6991 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/15c3cb1ae5d94446b8569671c68c6991 2023-04-16 20:55:00,039 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b70a2bdfb77b426d89a06ef273da1391 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b70a2bdfb77b426d89a06ef273da1391 2023-04-16 20:55:00,040 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3785df75078e40e09fbd087796fbe436 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3785df75078e40e09fbd087796fbe436 2023-04-16 20:55:00,041 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/22143937bcfa466e80bcde0629def12c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/22143937bcfa466e80bcde0629def12c 2023-04-16 20:55:00,041 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/54bef9c99b7d432dbec41c413c0a7d2d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/54bef9c99b7d432dbec41c413c0a7d2d 2023-04-16 20:55:00,042 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6d21bdd5b83a438eb4031c0a48c49127 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6d21bdd5b83a438eb4031c0a48c49127 2023-04-16 20:55:00,043 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/ae7430473601455b986a7f0816447975 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/ae7430473601455b986a7f0816447975 2023-04-16 20:55:00,044 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9029c50297734ba4b382fdcada3bc0f7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9029c50297734ba4b382fdcada3bc0f7 2023-04-16 20:55:00,045 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9d564851f026448b8cd203420b58e8d6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9d564851f026448b8cd203420b58e8d6 2023-04-16 20:55:00,046 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3ef3288a951c4eaaa463ad41b3aadda2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/3ef3288a951c4eaaa463ad41b3aadda2 2023-04-16 20:55:00,047 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4bde20bbc2b04474b174a9633837ba55 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4bde20bbc2b04474b174a9633837ba55 2023-04-16 20:55:00,048 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/cb62e311987443219729f6beab1a701b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/cb62e311987443219729f6beab1a701b 2023-04-16 20:55:00,049 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9b245478c9f843f38c18247b887a9775 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/9b245478c9f843f38c18247b887a9775 2023-04-16 20:55:00,050 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5f7b47f150474487a9415b66a3d2119d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/5f7b47f150474487a9415b66a3d2119d 2023-04-16 20:55:00,051 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4dee25723b384b81a47efa5241c625c3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/4dee25723b384b81a47efa5241c625c3 2023-04-16 20:55:00,051 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6014638af9ff4a80875664b9798d92de to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/6014638af9ff4a80875664b9798d92de 2023-04-16 20:55:00,052 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/89c65a6488a94af79bdeb27ef8883278 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/89c65a6488a94af79bdeb27ef8883278 2023-04-16 20:55:00,053 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b7316eb1135d49da91c050173585fd24 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/b7316eb1135d49da91c050173585fd24 2023-04-16 20:55:00,056 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4f4f6cb2cb314c7cb74d343329c5031b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/158271bdb65547049b0cc26d05b0db7f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/71147480450c43d7aee0a3136c6c44b5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e7aef42bfeff48799f352459cd732e46, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d7b79ead3ba416b8e8ab8b6b13d24fd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/01cee1b27c2b4ebf96c89ce756d51baa, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9f11e8f8aa0349468bf8c4886e75229e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/3ffc58a8be4b4bd59d1a2d21d1914da6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/29a9710ee5f246db84c2e55852c37984, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/784866c024ab49cfbe1306f361e6860c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/99dc69272e6a43de9224da7bdb713e53, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9e9f5204425c48a5a13556beeb6e608e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e28b848c40ce44bd93d1f9aa7725f3d3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/1402ecebf2d447a69143f0adcfea9449, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7e33a6ece6ae440cbf7210413e9b6345, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/8a51eb8ab7b94d16a5008f8127da78b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7a3a6fc02b684d20a514f50e504231e9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/81f48d08ecdb4d6ca4637edfc76e378d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/d67ce60d4fff4fdd98ffcdf05b4c790b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/45856d3e4f82427e878dede4d8542f52, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/ad3c543236dd4ecfbc90f01d07160813, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4fd97a79ebac4cb0ab66d87cc036bb9d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/334a9138b04e466f843ff36bee535e8a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/59fd61dfe54f45c68a0681791756606c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/eba266c97c09476c9ae54142ce41929e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d525251db7a4546a413612ac570b09c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/5e65e3ed6e814788a6d7f9583efc613d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/39dcd9e424dd4f1c93aae24150e15364, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/b9c924945ad44822b0d7012004f6859d] to archive 2023-04-16 20:55:00,057 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:00,058 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4f4f6cb2cb314c7cb74d343329c5031b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4f4f6cb2cb314c7cb74d343329c5031b 2023-04-16 20:55:00,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/158271bdb65547049b0cc26d05b0db7f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/158271bdb65547049b0cc26d05b0db7f 2023-04-16 20:55:00,060 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/71147480450c43d7aee0a3136c6c44b5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/71147480450c43d7aee0a3136c6c44b5 2023-04-16 20:55:00,060 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e7aef42bfeff48799f352459cd732e46 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e7aef42bfeff48799f352459cd732e46 2023-04-16 20:55:00,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d7b79ead3ba416b8e8ab8b6b13d24fd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d7b79ead3ba416b8e8ab8b6b13d24fd 2023-04-16 20:55:00,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/01cee1b27c2b4ebf96c89ce756d51baa to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/01cee1b27c2b4ebf96c89ce756d51baa 2023-04-16 20:55:00,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9f11e8f8aa0349468bf8c4886e75229e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9f11e8f8aa0349468bf8c4886e75229e 2023-04-16 20:55:00,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/3ffc58a8be4b4bd59d1a2d21d1914da6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/3ffc58a8be4b4bd59d1a2d21d1914da6 2023-04-16 20:55:00,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/29a9710ee5f246db84c2e55852c37984 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/29a9710ee5f246db84c2e55852c37984 2023-04-16 20:55:00,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/784866c024ab49cfbe1306f361e6860c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/784866c024ab49cfbe1306f361e6860c 2023-04-16 20:55:00,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/99dc69272e6a43de9224da7bdb713e53 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/99dc69272e6a43de9224da7bdb713e53 2023-04-16 20:55:00,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9e9f5204425c48a5a13556beeb6e608e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/9e9f5204425c48a5a13556beeb6e608e 2023-04-16 20:55:00,069 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e28b848c40ce44bd93d1f9aa7725f3d3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/e28b848c40ce44bd93d1f9aa7725f3d3 2023-04-16 20:55:00,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/1402ecebf2d447a69143f0adcfea9449 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/1402ecebf2d447a69143f0adcfea9449 2023-04-16 20:55:00,071 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7e33a6ece6ae440cbf7210413e9b6345 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7e33a6ece6ae440cbf7210413e9b6345 2023-04-16 20:55:00,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/8a51eb8ab7b94d16a5008f8127da78b0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/8a51eb8ab7b94d16a5008f8127da78b0 2023-04-16 20:55:00,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7a3a6fc02b684d20a514f50e504231e9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/7a3a6fc02b684d20a514f50e504231e9 2023-04-16 20:55:00,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/81f48d08ecdb4d6ca4637edfc76e378d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/81f48d08ecdb4d6ca4637edfc76e378d 2023-04-16 20:55:00,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/d67ce60d4fff4fdd98ffcdf05b4c790b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/d67ce60d4fff4fdd98ffcdf05b4c790b 2023-04-16 20:55:00,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/45856d3e4f82427e878dede4d8542f52 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/45856d3e4f82427e878dede4d8542f52 2023-04-16 20:55:00,076 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/ad3c543236dd4ecfbc90f01d07160813 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/ad3c543236dd4ecfbc90f01d07160813 2023-04-16 20:55:00,076 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4fd97a79ebac4cb0ab66d87cc036bb9d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/4fd97a79ebac4cb0ab66d87cc036bb9d 2023-04-16 20:55:00,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/334a9138b04e466f843ff36bee535e8a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/334a9138b04e466f843ff36bee535e8a 2023-04-16 20:55:00,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/59fd61dfe54f45c68a0681791756606c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/59fd61dfe54f45c68a0681791756606c 2023-04-16 20:55:00,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/eba266c97c09476c9ae54142ce41929e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/eba266c97c09476c9ae54142ce41929e 2023-04-16 20:55:00,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d525251db7a4546a413612ac570b09c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/2d525251db7a4546a413612ac570b09c 2023-04-16 20:55:00,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/5e65e3ed6e814788a6d7f9583efc613d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/5e65e3ed6e814788a6d7f9583efc613d 2023-04-16 20:55:00,081 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/39dcd9e424dd4f1c93aae24150e15364 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/39dcd9e424dd4f1c93aae24150e15364 2023-04-16 20:55:00,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/b9c924945ad44822b0d7012004f6859d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/b9c924945ad44822b0d7012004f6859d 2023-04-16 20:55:00,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/recovered.edits/1612.seqid, newMaxSeqId=1612, maxSeqId=1 2023-04-16 20:55:00,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda. 2023-04-16 20:55:00,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 968ac8675f340fdac83d49b0dce63dda: 2023-04-16 20:55:00,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:55:00,088 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=968ac8675f340fdac83d49b0dce63dda, regionState=CLOSED 2023-04-16 20:55:00,088 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678500088"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678500088"}]},"ts":"1681678500088"} 2023-04-16 20:55:00,090 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-16 20:55:00,090 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 968ac8675f340fdac83d49b0dce63dda, server=jenkins-hbase4.apache.org,41211,1681678416093 in 1.5020 sec 2023-04-16 20:55:00,091 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-16 20:55:00,091 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=968ac8675f340fdac83d49b0dce63dda, UNASSIGN in 1.5060 sec 2023-04-16 20:55:00,091 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678500091"}]},"ts":"1681678500091"} 2023-04-16 20:55:00,092 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 20:55:00,094 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 20:55:00,095 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5150 sec 2023-04-16 20:55:00,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 20:55:00,685 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-16 20:55:00,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 20:55:00,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:00,688 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:00,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-16 20:55:00,688 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:00,692 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:55:00,694 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/recovered.edits] 2023-04-16 20:55:00,697 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4419fecf3f2d4a2fb0827d6e38c8ccad to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/4419fecf3f2d4a2fb0827d6e38c8ccad 2023-04-16 20:55:00,698 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/a9864f8968df4a41a83aa386307257e2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/a9864f8968df4a41a83aa386307257e2 2023-04-16 20:55:00,699 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/eccac1d3abff40e5ace2a06a9e18d551 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/A/eccac1d3abff40e5ace2a06a9e18d551 2023-04-16 20:55:00,701 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/0a49b9c7822342689fb344ad16b9af68 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/0a49b9c7822342689fb344ad16b9af68 2023-04-16 20:55:00,702 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/284f71bc4af24681ab39b37b725b020a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/284f71bc4af24681ab39b37b725b020a 2023-04-16 20:55:00,703 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/d0ed9047cba0401d9c07d9cc01efe4b3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/B/d0ed9047cba0401d9c07d9cc01efe4b3 2023-04-16 20:55:00,705 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/05a0d6a5f4e14eb48e365b9aa8c09ae5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/05a0d6a5f4e14eb48e365b9aa8c09ae5 2023-04-16 20:55:00,706 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/224e296496a24e249df86d70edf5f068 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/224e296496a24e249df86d70edf5f068 2023-04-16 20:55:00,708 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/726a31a98a6344f18c7c079efa36b13b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/726a31a98a6344f18c7c079efa36b13b 2023-04-16 20:55:00,709 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/924d41709fd344769ff0a0b49af814e5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/C/924d41709fd344769ff0a0b49af814e5 2023-04-16 20:55:00,712 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/recovered.edits/1612.seqid to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda/recovered.edits/1612.seqid 2023-04-16 20:55:00,712 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/968ac8675f340fdac83d49b0dce63dda 2023-04-16 20:55:00,713 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:55:00,715 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:00,716 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 20:55:00,717 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 20:55:00,718 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:00,718 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 20:55:00,718 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681678500718"}]},"ts":"9223372036854775807"} 2023-04-16 20:55:00,720 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 20:55:00,720 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 968ac8675f340fdac83d49b0dce63dda, NAME => 'TestAcidGuarantees,,1681678474353.968ac8675f340fdac83d49b0dce63dda.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 20:55:00,720 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 20:55:00,720 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681678500720"}]},"ts":"9223372036854775807"} 2023-04-16 20:55:00,722 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 20:55:00,725 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:00,725 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 40 msec 2023-04-16 20:55:00,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-16 20:55:00,789 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-16 20:55:00,801 INFO [Listener at localhost/45179] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 233), OpenFileDescriptor=542 (was 574), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 413), ProcessCount=169 (was 169), AvailableMemoryMB=491 (was 532) 2023-04-16 20:55:00,811 INFO [Listener at localhost/45179] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=542, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=169, AvailableMemoryMB=490 2023-04-16 20:55:00,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 20:55:00,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:55:00,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:00,815 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 20:55:00,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-16 20:55:00,815 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 20:55:00,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 20:55:00,817 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:00,817 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1 empty. 2023-04-16 20:55:00,817 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:00,817 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:55:00,824 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 20:55:00,825 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1e76395fa47d06dccc999b09513e8aa1, NAME => 'TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp 2023-04-16 20:55:00,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 20:55:01,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 20:55:01,234 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:55:01,234 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1e76395fa47d06dccc999b09513e8aa1, disabling compactions & flushes 2023-04-16 20:55:01,234 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:01,234 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:01,234 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. after waiting 0 ms 2023-04-16 20:55:01,234 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:01,234 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:01,234 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:01,236 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 20:55:01,237 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678501237"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678501237"}]},"ts":"1681678501237"} 2023-04-16 20:55:01,238 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 20:55:01,238 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 20:55:01,238 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678501238"}]},"ts":"1681678501238"} 2023-04-16 20:55:01,239 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 20:55:01,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, ASSIGN}] 2023-04-16 20:55:01,243 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, ASSIGN 2023-04-16 20:55:01,244 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=false 2023-04-16 20:55:01,395 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=1e76395fa47d06dccc999b09513e8aa1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:01,395 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678501395"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678501395"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678501395"}]},"ts":"1681678501395"} 2023-04-16 20:55:01,396 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:55:01,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 20:55:01,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:01,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1e76395fa47d06dccc999b09513e8aa1, NAME => 'TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:55:01,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:01,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:55:01,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:01,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:01,552 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:01,554 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A 2023-04-16 20:55:01,554 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A 2023-04-16 20:55:01,554 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:01,554 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1e76395fa47d06dccc999b09513e8aa1 columnFamilyName A 2023-04-16 20:55:01,554 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(310): Store=1e76395fa47d06dccc999b09513e8aa1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:01,555 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:01,556 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B 2023-04-16 20:55:01,556 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B 2023-04-16 20:55:01,556 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:01,556 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1e76395fa47d06dccc999b09513e8aa1 columnFamilyName B 2023-04-16 20:55:01,557 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(310): Store=1e76395fa47d06dccc999b09513e8aa1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:01,557 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:01,558 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C 2023-04-16 20:55:01,558 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C 2023-04-16 20:55:01,558 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:01,558 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1e76395fa47d06dccc999b09513e8aa1 columnFamilyName C 2023-04-16 20:55:01,559 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(310): Store=1e76395fa47d06dccc999b09513e8aa1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:01,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:01,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:01,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:01,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:55:01,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:01,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 20:55:01,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1e76395fa47d06dccc999b09513e8aa1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60997841, jitterRate=-0.09106133878231049}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:55:01,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:01,565 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., pid=37, masterSystemTime=1681678501547 2023-04-16 20:55:01,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:01,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:01,566 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=1e76395fa47d06dccc999b09513e8aa1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:01,566 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678501566"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678501566"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678501566"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678501566"}]},"ts":"1681678501566"} 2023-04-16 20:55:01,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-16 20:55:01,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 in 171 msec 2023-04-16 20:55:01,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-16 20:55:01,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, ASSIGN in 326 msec 2023-04-16 20:55:01,570 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 20:55:01,570 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678501570"}]},"ts":"1681678501570"} 2023-04-16 20:55:01,571 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 20:55:01,572 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 20:55:01,573 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 759 msec 2023-04-16 20:55:01,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 20:55:01,919 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-16 20:55:01,919 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:01,925 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:01,926 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:01,928 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46726, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:01,929 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 20:55:01,930 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 20:55:01,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 20:55:01,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:55:01,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:02,341 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 20:55:02,342 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 20:55:02,342 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 20:55:02,344 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-16 20:55:02,346 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, REOPEN/MOVE}] 2023-04-16 20:55:02,346 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, REOPEN/MOVE 2023-04-16 20:55:02,347 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1e76395fa47d06dccc999b09513e8aa1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:02,347 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678502347"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678502347"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678502347"}]},"ts":"1681678502347"} 2023-04-16 20:55:02,348 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:55:02,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1e76395fa47d06dccc999b09513e8aa1, disabling compactions & flushes 2023-04-16 20:55:02,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:02,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:02,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. after waiting 0 ms 2023-04-16 20:55:02,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:02,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-16 20:55:02,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:02,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:02,504 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 1e76395fa47d06dccc999b09513e8aa1 to self. 2023-04-16 20:55:02,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,505 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1e76395fa47d06dccc999b09513e8aa1, regionState=CLOSED 2023-04-16 20:55:02,505 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678502505"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678502505"}]},"ts":"1681678502505"} 2023-04-16 20:55:02,507 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-16 20:55:02,507 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 in 158 msec 2023-04-16 20:55:02,507 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=true 2023-04-16 20:55:02,658 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1e76395fa47d06dccc999b09513e8aa1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:02,658 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678502658"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678502658"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678502658"}]},"ts":"1681678502658"} 2023-04-16 20:55:02,659 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:55:02,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:02,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1e76395fa47d06dccc999b09513e8aa1, NAME => 'TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:55:02,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:55:02,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,815 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,816 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A 2023-04-16 20:55:02,816 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A 2023-04-16 20:55:02,817 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:02,817 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1e76395fa47d06dccc999b09513e8aa1 columnFamilyName A 2023-04-16 20:55:02,818 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(310): Store=1e76395fa47d06dccc999b09513e8aa1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:02,818 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,819 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B 2023-04-16 20:55:02,819 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B 2023-04-16 20:55:02,819 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:02,819 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1e76395fa47d06dccc999b09513e8aa1 columnFamilyName B 2023-04-16 20:55:02,819 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(310): Store=1e76395fa47d06dccc999b09513e8aa1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:02,819 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,820 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C 2023-04-16 20:55:02,820 DEBUG [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C 2023-04-16 20:55:02,820 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:02,820 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1e76395fa47d06dccc999b09513e8aa1 columnFamilyName C 2023-04-16 20:55:02,821 INFO [StoreOpener-1e76395fa47d06dccc999b09513e8aa1-1] regionserver.HStore(310): Store=1e76395fa47d06dccc999b09513e8aa1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:02,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:02,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:55:02,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1e76395fa47d06dccc999b09513e8aa1; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60133808, jitterRate=-0.10393643379211426}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:55:02,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:02,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., pid=42, masterSystemTime=1681678502810 2023-04-16 20:55:02,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:02,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:02,827 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1e76395fa47d06dccc999b09513e8aa1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:02,827 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678502827"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678502827"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678502827"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678502827"}]},"ts":"1681678502827"} 2023-04-16 20:55:02,829 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-16 20:55:02,829 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 in 169 msec 2023-04-16 20:55:02,830 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-16 20:55:02,830 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, REOPEN/MOVE in 483 msec 2023-04-16 20:55:02,831 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-16 20:55:02,831 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 486 msec 2023-04-16 20:55:02,833 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 901 msec 2023-04-16 20:55:02,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-16 20:55:02,834 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,842 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,842 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,848 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,848 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,854 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,854 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,859 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,859 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,864 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,864 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,867 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,867 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,873 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,873 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,879 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,879 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x0eb8f142 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,882 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e079da1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,883 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x5ce79633 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:02,886 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@159c5e0c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:02,894 DEBUG [hconnection-0x4b7b0f3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,894 DEBUG [hconnection-0x33aa2824-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,894 DEBUG [hconnection-0x475d6b4c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,895 DEBUG [hconnection-0x414e4293-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,895 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46734, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,896 DEBUG [hconnection-0x53d4b5c4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,896 DEBUG [hconnection-0x6693b4d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,897 DEBUG [hconnection-0x5d2ee595-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:02,897 DEBUG [hconnection-0x76e9964c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,897 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46744, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,897 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46736, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,897 DEBUG [hconnection-0x51c602c9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,899 DEBUG [hconnection-0x376b1008-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:02,901 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,903 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46780, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,903 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,903 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46746, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,906 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,906 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,906 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:02,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:02,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:02,910 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:02,910 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:02,910 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:02,910 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:02,911 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:02,911 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:02,914 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:02,914 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:02,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:02,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:02,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:02,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:02,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:02,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:02,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:02,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:02,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:02,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:02,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:02,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:02,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:02,917 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:02,917 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:02,918 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-16 20:55:02,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:02,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:02,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:02,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678562934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:02,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:02,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678562936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:02,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678502930, max timestamp=1681678502939Num uniques 859; 2023-04-16 20:55:02,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678502930, max timestamp=1681678502939Num uniques 859; 2023-04-16 20:55:02,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:55:02,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:55:02,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678502930, max timestamp=1681678502939Num uniques 859; 2023-04-16 20:55:02,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:55:02,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:02,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678562958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:02,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:02,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678562959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:02,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:02,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678562963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678563042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678563042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678563064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678563065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678563065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678563249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678563250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678563268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678563269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678563269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,339 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304165df7035025a446ad88a22490aa82529a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165df7035025a446ad88a22490aa82529a 2023-04-16 20:55:03,345 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/ccf630370f11466bb69be68c464c891e 2023-04-16 20:55:03,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678563552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678563554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678563574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678563575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:03,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678563576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:03,759 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/192ce44a06534468832fe1cbad1cf25e 2023-04-16 20:55:04,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:04,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678564058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:04,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678564058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:04,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678564079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:04,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678564082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:04,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678564082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,168 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/ecc9aa2b9f0542bba8d1e273678acad3 2023-04-16 20:55:04,173 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/ccf630370f11466bb69be68c464c891e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ccf630370f11466bb69be68c464c891e 2023-04-16 20:55:04,176 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ccf630370f11466bb69be68c464c891e, entries=150, sequenceid=15, filesize=25.1 K 2023-04-16 20:55:04,177 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/192ce44a06534468832fe1cbad1cf25e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/192ce44a06534468832fe1cbad1cf25e 2023-04-16 20:55:04,180 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/192ce44a06534468832fe1cbad1cf25e, entries=150, sequenceid=15, filesize=11.6 K 2023-04-16 20:55:04,181 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/ecc9aa2b9f0542bba8d1e273678acad3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ecc9aa2b9f0542bba8d1e273678acad3 2023-04-16 20:55:04,183 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ecc9aa2b9f0542bba8d1e273678acad3, entries=150, sequenceid=15, filesize=11.6 K 2023-04-16 20:55:04,184 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1e76395fa47d06dccc999b09513e8aa1 in 1266ms, sequenceid=15, compaction requested=false 2023-04-16 20:55:04,184 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:04,184 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:04,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:04,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:04,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:04,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:04,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dedd86e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:04,187 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:04,187 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:04,188 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:04,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:04,189 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:04,189 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:04,189 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:04,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:04,190 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:04,192 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:04,192 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:04,192 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:04,192 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:04,192 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:04,194 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:04,195 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:04,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:04,195 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:04,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:04,195 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:04,196 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,201 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:04,201 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:04,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:04,201 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:04,201 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:04,202 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:04,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:04,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:04,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:04,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:04,202 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,202 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:04,202 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:04,202 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:04,202 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:04,202 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:04,302 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:04,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:04,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:04,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:04,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:04,306 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:04,306 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:04,306 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:04,306 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:04,306 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:04,306 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:04,307 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,307 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:04,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:04,308 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:04,308 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:04,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:04,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:04,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:04,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:04,309 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:04,309 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:04,309 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-16 20:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:04,331 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304165d2ed8fd478c45178fad1a98a1caf5d4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165d2ed8fd478c45178fad1a98a1caf5d4 2023-04-16 20:55:04,743 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/ec919528ef42494cbb52645f8c0a7367 2023-04-16 20:55:04,754 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/b1bce80ecfa84eaaacc746d79ca83867 2023-04-16 20:55:05,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:05,069 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:05,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678565082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678565085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678565088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678565089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678565095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,163 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/0508310058ca4c9ca671ffaa36ab4dac 2023-04-16 20:55:05,167 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/ec919528ef42494cbb52645f8c0a7367 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ec919528ef42494cbb52645f8c0a7367 2023-04-16 20:55:05,171 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ec919528ef42494cbb52645f8c0a7367, entries=150, sequenceid=49, filesize=25.1 K 2023-04-16 20:55:05,172 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/b1bce80ecfa84eaaacc746d79ca83867 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/b1bce80ecfa84eaaacc746d79ca83867 2023-04-16 20:55:05,175 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/b1bce80ecfa84eaaacc746d79ca83867, entries=150, sequenceid=49, filesize=11.6 K 2023-04-16 20:55:05,175 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/0508310058ca4c9ca671ffaa36ab4dac as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/0508310058ca4c9ca671ffaa36ab4dac 2023-04-16 20:55:05,178 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/0508310058ca4c9ca671ffaa36ab4dac, entries=150, sequenceid=49, filesize=11.6 K 2023-04-16 20:55:05,179 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 1e76395fa47d06dccc999b09513e8aa1 in 870ms, sequenceid=49, compaction requested=false 2023-04-16 20:55:05,179 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:05,179 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:05,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,182 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,182 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,182 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:05,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ea42a1[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:05,183 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:05,184 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,185 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:05,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:05,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:05,185 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,185 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:05,186 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:05,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:05,189 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:05,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:05,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:05,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:05,189 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:05,190 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,190 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,190 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,191 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,193 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:05,195 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:55:05,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:05,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:05,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:05,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:05,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:05,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:05,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,200 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,200 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:05,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:05,200 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,200 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,200 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:05,200 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,200 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:05,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678565249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678565250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,300 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:05,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:05,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:05,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:05,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:05,304 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:05,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:05,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:05,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,305 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,305 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,306 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,306 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:05,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:05,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:05,307 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,307 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:05,307 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:05,307 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,308 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:05,308 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:05,308 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,308 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:05,308 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:05,309 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,310 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,310 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:05,311 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:05,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477bb79a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:05,311 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,312 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:05,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:05,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:05,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:05,315 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:05,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:05,317 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:05,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:05,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:05,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:05,318 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:05,319 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,319 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,319 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,320 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,321 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,323 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,323 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,323 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,323 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:05,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:05,323 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,324 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:05,324 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:05,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678565354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678565356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,424 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:05,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:05,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:05,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:05,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:05,427 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:05,427 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,427 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:05,427 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:05,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,429 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,429 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:05,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:05,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:05,431 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,431 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:05,431 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:05,431 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:05,433 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:05,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e76b47[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:05,436 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:05,436 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,437 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,438 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:05,438 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:05,438 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,438 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:05,438 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:05,439 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:05,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,439 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,439 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:05,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:05,441 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:05,441 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:05,441 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:05,441 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:05,442 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:05,443 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,443 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,443 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,444 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,451 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:05,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:05,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:05,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,451 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:05,451 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:05,552 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:05,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:05,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:05,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:05,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:05,555 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:05,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:05,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:05,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,557 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:05,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:05,558 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,558 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:05,559 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:05,559 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:05,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,560 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,560 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,560 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:05,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47baeb1d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:05,562 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:05,562 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678565558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678565560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,563 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,563 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,563 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:05,563 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:05,563 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:05,563 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:05,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,566 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,566 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,566 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:05,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:05,568 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:05,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:05,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:05,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:05,569 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:05,570 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,571 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,573 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,574 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,574 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,574 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,574 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:05,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:05,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,575 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:05,576 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:05,609 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041669b07a7bd0994560bcbc7ded23c5b0ca to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041669b07a7bd0994560bcbc7ded23c5b0ca 2023-04-16 20:55:05,609 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 20:55:05,613 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/ef0d4235622d4ac38aae362009f5e701 2023-04-16 20:55:05,676 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:05,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:05,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:05,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:05,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:05,679 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:05,679 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,679 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:05,679 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:05,679 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,679 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,681 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:05,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:05,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:05,682 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,682 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:05,682 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:05,683 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,683 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:05,683 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:05,683 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,683 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:05,683 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:05,686 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,686 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,686 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,686 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:05,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65e75601[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:05,691 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:05,691 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:05,692 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,692 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:05,693 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,693 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:05,693 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,693 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:05,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:05,696 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:05,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:05,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:05,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:05,696 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:05,697 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,697 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,697 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,698 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,699 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,703 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,703 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,703 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,703 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,703 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,703 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,703 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:05,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:05,703 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,704 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,704 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,704 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:05,704 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:05,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,804 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:05,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:05,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:05,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:05,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:05,807 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:05,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:05,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:05,808 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,808 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,809 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:05,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:05,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:05,811 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,811 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:05,811 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:05,811 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:05,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:05,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3aea40[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:05,814 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:05,814 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,815 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,815 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:05,815 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:05,815 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:05,816 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,816 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:05,818 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,818 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,818 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,818 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:05,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:05,820 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:05,820 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:05,820 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:05,820 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:05,820 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:05,822 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,822 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,822 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,826 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,826 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,826 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:05,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:05,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,826 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,826 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,826 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,827 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:05,827 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,827 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:05,827 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,827 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,827 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678565864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:05,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678565867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,927 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:05,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:05,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:05,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:05,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:05,930 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:05,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:05,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:05,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,935 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,935 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,935 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,935 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:05,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:05,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:05,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:05,936 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:05,936 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:05,936 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:05,937 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,937 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:05,937 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:05,937 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:05,939 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,939 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,939 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,939 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:05,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:05,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70881bb3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:05,941 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:05,941 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,942 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,942 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,942 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:05,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:05,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:05,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:05,943 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:05,943 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,943 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,943 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:05,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:05,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:05,946 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:05,946 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:05,946 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:05,946 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:05,946 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:05,948 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,949 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,949 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:05,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,949 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:05,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:05,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,950 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,952 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,952 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:05,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:05,952 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:05,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:05,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:05,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:05,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:05,952 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:05,953 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:05,953 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:05,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:05,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:05,953 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:05,953 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:05,953 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,953 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:05,953 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/722361040ffa4630816c3e5664ccf92f 2023-04-16 20:55:06,053 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:06,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:06,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:06,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:06,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:06,056 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:06,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:06,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:06,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,058 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:06,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:06,058 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,058 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:06,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:06,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:06,060 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,060 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:06,060 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:06,060 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:06,061 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,061 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,061 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:06,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60fae4bc[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:06,063 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:06,063 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,066 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,066 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:06,066 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:06,066 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:06,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:06,067 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:06,067 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:06,067 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,067 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:06,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:06,069 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:06,069 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:06,070 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:06,070 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:06,070 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:06,071 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,071 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,071 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,071 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,073 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,075 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,075 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,075 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,075 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,075 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,075 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:06,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:06,075 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:06,075 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,076 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:06,076 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:06,082 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:06,082 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:06,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:06,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:06,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:06,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:06,176 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:06,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:06,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:06,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:06,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:06,179 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:06,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:06,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:06,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:06,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:06,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:06,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:06,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:06,182 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:06,182 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,182 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:06,182 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:06,183 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:06,183 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,183 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:06,183 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:06,183 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,183 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:06,183 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:06,183 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,183 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:06,183 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,184 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:06,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f72e1fc[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:06,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:06,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,187 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:06,187 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,187 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:06,188 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:06,188 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,188 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,188 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:06,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:06,190 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:06,190 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:06,190 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:06,190 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:06,190 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:06,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:06,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,194 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,195 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:06,197 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:06,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:06,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:06,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:06,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:06,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,197 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:06,197 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:06,197 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:06,298 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:06,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:06,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:06,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:06,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:06,301 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:06,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:06,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:06,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,302 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:06,302 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:06,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,303 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,303 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:06,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:06,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:06,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:06,304 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,304 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:06,304 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:06,304 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:06,304 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:06,309 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:06,311 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:06,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@173274b8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:06,311 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,312 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:06,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:06,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:06,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:06,313 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,313 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:06,313 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,313 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:06,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:06,316 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:06,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:06,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:06,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:06,316 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:06,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:06,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,318 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,319 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:06,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:06,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:06,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:06,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:06,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:06,322 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:06,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,323 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:06,323 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:06,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:06,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:06,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678566369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:06,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678566372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,423 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:06,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:06,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:06,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:06,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:06,428 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:06,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:06,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:06,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:06,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:06,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:06,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,430 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:06,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:06,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:06,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:06,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:06,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:06,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,431 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:06,432 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:06,432 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:06,432 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,432 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:06,432 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:06,432 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:06,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:06,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:06,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:06,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,434 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/805e29240d6e47e9a6ce0dc55eb56017 2023-04-16 20:55:06,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:06,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27d7b06e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:06,436 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:06,436 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,438 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,438 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:06,438 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,438 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:06,438 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:06,438 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:06,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/ef0d4235622d4ac38aae362009f5e701 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ef0d4235622d4ac38aae362009f5e701 2023-04-16 20:55:06,441 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:06,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:06,444 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:06,444 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:06,444 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:06,444 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:06,444 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:06,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ef0d4235622d4ac38aae362009f5e701, entries=200, sequenceid=61, filesize=31.9 K 2023-04-16 20:55:06,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/722361040ffa4630816c3e5664ccf92f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/722361040ffa4630816c3e5664ccf92f 2023-04-16 20:55:06,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:06,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:06,446 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/722361040ffa4630816c3e5664ccf92f, entries=150, sequenceid=61, filesize=11.6 K 2023-04-16 20:55:06,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:06,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:06,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/805e29240d6e47e9a6ce0dc55eb56017 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/805e29240d6e47e9a6ce0dc55eb56017 2023-04-16 20:55:06,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:06,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:06,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:06,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,452 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:06,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:06,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:06,452 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:06,452 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:06,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:06,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:06,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:06,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/805e29240d6e47e9a6ce0dc55eb56017, entries=150, sequenceid=61, filesize=11.6 K 2023-04-16 20:55:06,455 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1e76395fa47d06dccc999b09513e8aa1 in 1260ms, sequenceid=61, compaction requested=true 2023-04-16 20:55:06,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:06,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:06,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:06,455 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:06,455 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:06,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:06,456 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:06,456 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:06,456 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/A is initiating minor compaction (all files) 2023-04-16 20:55:06,457 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/B is initiating minor compaction (all files) 2023-04-16 20:55:06,457 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/A in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,457 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/B in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,457 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ccf630370f11466bb69be68c464c891e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ec919528ef42494cbb52645f8c0a7367, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ef0d4235622d4ac38aae362009f5e701] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=82.2 K 2023-04-16 20:55:06,457 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/192ce44a06534468832fe1cbad1cf25e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/b1bce80ecfa84eaaacc746d79ca83867, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/722361040ffa4630816c3e5664ccf92f] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=34.8 K 2023-04-16 20:55:06,457 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ccf630370f11466bb69be68c464c891e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681678502906 2023-04-16 20:55:06,457 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 192ce44a06534468832fe1cbad1cf25e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681678502906 2023-04-16 20:55:06,458 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ec919528ef42494cbb52645f8c0a7367, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681678502950 2023-04-16 20:55:06,458 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b1bce80ecfa84eaaacc746d79ca83867, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681678502950 2023-04-16 20:55:06,458 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ef0d4235622d4ac38aae362009f5e701, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681678505069 2023-04-16 20:55:06,458 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 722361040ffa4630816c3e5664ccf92f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681678505069 2023-04-16 20:55:06,471 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#B#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:06,475 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#A#compaction#338 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:06,498 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/ee2c9f0cf84a41488d7ba068bd9fe39b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/ee2c9f0cf84a41488d7ba068bd9fe39b 2023-04-16 20:55:06,506 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/B of 1e76395fa47d06dccc999b09513e8aa1 into ee2c9f0cf84a41488d7ba068bd9fe39b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:06,506 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:06,506 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/B, priority=13, startTime=1681678506455; duration=0sec 2023-04-16 20:55:06,506 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:06,506 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:06,507 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:06,507 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/C is initiating minor compaction (all files) 2023-04-16 20:55:06,508 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/C in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,508 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ecc9aa2b9f0542bba8d1e273678acad3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/0508310058ca4c9ca671ffaa36ab4dac, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/805e29240d6e47e9a6ce0dc55eb56017] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=34.8 K 2023-04-16 20:55:06,508 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ecc9aa2b9f0542bba8d1e273678acad3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681678502906 2023-04-16 20:55:06,508 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0508310058ca4c9ca671ffaa36ab4dac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681678502950 2023-04-16 20:55:06,509 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 805e29240d6e47e9a6ce0dc55eb56017, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681678505069 2023-04-16 20:55:06,514 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#C#compaction#340 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:06,526 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/f61c987d0c494cb88daefd84994db3bd as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f61c987d0c494cb88daefd84994db3bd 2023-04-16 20:55:06,530 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/C of 1e76395fa47d06dccc999b09513e8aa1 into f61c987d0c494cb88daefd84994db3bd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:06,530 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:06,530 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/C, priority=13, startTime=1681678506455; duration=0sec 2023-04-16 20:55:06,530 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:06,552 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:06,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:06,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:06,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:06,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:06,555 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:06,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:06,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:06,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,557 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:06,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:06,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:06,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:06,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:06,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:06,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:06,558 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:06,558 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:06,558 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 20:55:06,558 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:06,558 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:06,558 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:06,559 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:06,559 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:06,559 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:06,974 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416c96e1e2620224a89a39b56fe8539a721 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c96e1e2620224a89a39b56fe8539a721 2023-04-16 20:55:06,979 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/330a8a5f1286432a8cf7cab731681cf3 2023-04-16 20:55:06,997 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/956e8e9b2ebc4cd3acd741cd13d200b8 2023-04-16 20:55:07,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:07,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:07,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678567123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678567123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678567125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678567226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678567229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678567229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,312 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/9166094267124ca6acfe07c2b9c81c4a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9166094267124ca6acfe07c2b9c81c4a 2023-04-16 20:55:07,317 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/A of 1e76395fa47d06dccc999b09513e8aa1 into 9166094267124ca6acfe07c2b9c81c4a(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:07,317 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:07,318 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/A, priority=13, startTime=1681678506455; duration=0sec 2023-04-16 20:55:07,318 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:07,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678567375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678567380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,406 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/ccbaeefd841b47469632f8378e434d8e 2023-04-16 20:55:07,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/330a8a5f1286432a8cf7cab731681cf3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/330a8a5f1286432a8cf7cab731681cf3 2023-04-16 20:55:07,414 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/330a8a5f1286432a8cf7cab731681cf3, entries=150, sequenceid=83, filesize=25.1 K 2023-04-16 20:55:07,415 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/956e8e9b2ebc4cd3acd741cd13d200b8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/956e8e9b2ebc4cd3acd741cd13d200b8 2023-04-16 20:55:07,420 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/956e8e9b2ebc4cd3acd741cd13d200b8, entries=150, sequenceid=83, filesize=11.6 K 2023-04-16 20:55:07,420 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/ccbaeefd841b47469632f8378e434d8e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ccbaeefd841b47469632f8378e434d8e 2023-04-16 20:55:07,425 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ccbaeefd841b47469632f8378e434d8e, entries=150, sequenceid=83, filesize=11.6 K 2023-04-16 20:55:07,425 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1e76395fa47d06dccc999b09513e8aa1 in 867ms, sequenceid=83, compaction requested=false 2023-04-16 20:55:07,426 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:07,426 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,426 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:07,426 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:07,426 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,426 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:07,426 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:07,429 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,430 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,430 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,430 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:07,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10bf6f2c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:07,432 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:07,432 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,433 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:07,433 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,433 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:07,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:07,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,434 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:07,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:07,437 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:07,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:07,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:07,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:07,437 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:07,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,441 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,444 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,444 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,444 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,444 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:07,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,444 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:07,444 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,444 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,445 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:07,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:07,445 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:07,445 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:07,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:07,448 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:55:07,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:07,449 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:07,449 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:07,449 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:07,449 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:07,449 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:07,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:07,499 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041646049ab02c174aa8b83ae745e068f229 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041646049ab02c174aa8b83ae745e068f229 2023-04-16 20:55:07,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678567500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678567499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678567501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,546 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:07,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:07,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:07,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:07,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:07,548 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:07,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:07,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:07,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:07,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,550 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:07,550 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:07,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:07,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:07,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:07,552 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,552 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:07,552 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:07,552 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,554 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,554 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38ea01c6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:07,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:07,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:07,557 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:07,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:07,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:07,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:07,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:07,564 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:07,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:07,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:07,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:07,564 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:07,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,568 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,570 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:07,570 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:07,570 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:07,570 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,570 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:07,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:07,571 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:07,571 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:07,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:07,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678567611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678567611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678567611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,671 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:07,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:07,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:07,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:07,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:07,674 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:07,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:07,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:07,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:07,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,676 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:07,676 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:07,677 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,677 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:07,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:07,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:07,678 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,678 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:07,678 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:07,678 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:07,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:07,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:07,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7935d7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:07,682 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:07,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:07,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:07,682 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,682 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:07,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:07,686 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:07,686 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:07,686 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:07,686 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:07,686 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:07,688 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,688 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,688 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,689 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,690 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:07,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:07,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:07,692 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:07,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:07,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,693 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,693 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,693 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:07,693 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,693 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:07,693 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,693 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,793 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:07,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:07,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:07,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:07,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:07,797 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:07,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:07,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:07,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:07,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,799 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:07,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:07,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:07,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:07,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:07,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:07,800 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:07,800 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:07,800 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:07,800 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:07,800 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:07,800 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,801 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:07,801 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:07,801 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:07,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:07,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:07,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:07,802 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:07,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b7b6f6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:07,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:07,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,806 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:07,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:07,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:07,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:07,808 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:07,808 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,808 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:07,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:07,810 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:07,810 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:07,810 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:07,810 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:07,810 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:07,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,813 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,814 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:07,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,816 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:07,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:07,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:07,816 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:07,816 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:07,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678567815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678567815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:07,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678567816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,916 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/bff209a3621e4f1d8a8bd35616e14435 2023-04-16 20:55:07,917 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:07,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:07,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:07,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:07,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:07,920 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:07,920 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:07,920 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:07,920 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:07,922 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,922 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,925 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:07,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:07,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:07,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,926 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:07,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:07,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:07,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:07,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:07,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:07,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:07,940 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:07,940 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:07,940 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,940 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:07,940 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:07,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/fa2eff94ad30433b8815b5d30d990fff 2023-04-16 20:55:07,940 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,940 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:07,940 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:07,942 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,943 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1a7b03[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:07,944 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:07,944 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,945 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,946 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,946 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:07,948 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:07,951 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:07,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:07,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:07,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:07,951 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:07,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/90db54c4b766441a953bfea4fa29ddf9 2023-04-16 20:55:07,956 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,956 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,956 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:07,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/bff209a3621e4f1d8a8bd35616e14435 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/bff209a3621e4f1d8a8bd35616e14435 2023-04-16 20:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,958 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/bff209a3621e4f1d8a8bd35616e14435, entries=150, sequenceid=97, filesize=25.1 K 2023-04-16 20:55:07,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/fa2eff94ad30433b8815b5d30d990fff as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/fa2eff94ad30433b8815b5d30d990fff 2023-04-16 20:55:07,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:07,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:07,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:07,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:07,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:07,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:07,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:07,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:07,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:07,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:07,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:07,964 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:07,964 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:07,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:07,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/fa2eff94ad30433b8815b5d30d990fff, entries=100, sequenceid=97, filesize=9.3 K 2023-04-16 20:55:07,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/90db54c4b766441a953bfea4fa29ddf9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/90db54c4b766441a953bfea4fa29ddf9 2023-04-16 20:55:07,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/90db54c4b766441a953bfea4fa29ddf9, entries=100, sequenceid=97, filesize=9.3 K 2023-04-16 20:55:07,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1e76395fa47d06dccc999b09513e8aa1 in 523ms, sequenceid=97, compaction requested=true 2023-04-16 20:55:07,971 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:07,971 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:07,971 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:07,971 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:07,971 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:07,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:07,972 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33417 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:07,972 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77337 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:07,973 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/B is initiating minor compaction (all files) 2023-04-16 20:55:07,973 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/A is initiating minor compaction (all files) 2023-04-16 20:55:07,973 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/B in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,973 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/A in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:07,973 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/ee2c9f0cf84a41488d7ba068bd9fe39b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/956e8e9b2ebc4cd3acd741cd13d200b8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/fa2eff94ad30433b8815b5d30d990fff] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=32.6 K 2023-04-16 20:55:07,973 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9166094267124ca6acfe07c2b9c81c4a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/330a8a5f1286432a8cf7cab731681cf3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/bff209a3621e4f1d8a8bd35616e14435] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=75.5 K 2023-04-16 20:55:07,973 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9166094267124ca6acfe07c2b9c81c4a, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681678505069 2023-04-16 20:55:07,973 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ee2c9f0cf84a41488d7ba068bd9fe39b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681678505069 2023-04-16 20:55:07,974 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 330a8a5f1286432a8cf7cab731681cf3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681678505242 2023-04-16 20:55:07,974 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 956e8e9b2ebc4cd3acd741cd13d200b8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681678505242 2023-04-16 20:55:07,974 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting bff209a3621e4f1d8a8bd35616e14435, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678507122 2023-04-16 20:55:07,975 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting fa2eff94ad30433b8815b5d30d990fff, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678507122 2023-04-16 20:55:07,997 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#B#compaction#348 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:08,022 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#A#compaction#347 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:08,065 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:08,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:08,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:08,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:08,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:08,068 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:08,068 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:08,068 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:08,068 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:08,068 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:08,068 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:08,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:08,070 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:08,070 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,070 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:08,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:08,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:08,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:08,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:08,071 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:08,071 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:08,071 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-16 20:55:08,071 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:08,071 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:08,072 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:08,072 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:08,072 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:08,072 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:08,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:08,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:08,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678568138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678568141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678568142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678568245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678568250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678568252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,440 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/e7814a62d9ee43a38251dff6fc466d56 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/e7814a62d9ee43a38251dff6fc466d56 2023-04-16 20:55:08,444 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/B of 1e76395fa47d06dccc999b09513e8aa1 into e7814a62d9ee43a38251dff6fc466d56(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:08,444 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:08,444 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/B, priority=13, startTime=1681678507971; duration=0sec 2023-04-16 20:55:08,444 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:08,444 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:08,445 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33417 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:08,445 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/C is initiating minor compaction (all files) 2023-04-16 20:55:08,445 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/C in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:08,445 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f61c987d0c494cb88daefd84994db3bd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ccbaeefd841b47469632f8378e434d8e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/90db54c4b766441a953bfea4fa29ddf9] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=32.6 K 2023-04-16 20:55:08,446 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting f61c987d0c494cb88daefd84994db3bd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681678505069 2023-04-16 20:55:08,446 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ccbaeefd841b47469632f8378e434d8e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681678505242 2023-04-16 20:55:08,447 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 90db54c4b766441a953bfea4fa29ddf9, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678507122 2023-04-16 20:55:08,452 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#C#compaction#350 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:08,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678568449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678568455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678568455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,487 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416b91ccaa65ef54c36b1a52498da71fe23 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b91ccaa65ef54c36b1a52498da71fe23 2023-04-16 20:55:08,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678568756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678568761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:08,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678568761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:08,866 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/c16740a9b5104898a6595c105d17585a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c16740a9b5104898a6595c105d17585a 2023-04-16 20:55:08,870 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/C of 1e76395fa47d06dccc999b09513e8aa1 into c16740a9b5104898a6595c105d17585a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:08,870 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:08,870 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/C, priority=13, startTime=1681678507972; duration=0sec 2023-04-16 20:55:08,870 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:08,895 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=116, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/78e03149066347ff9ef34f9c09f4a3ff 2023-04-16 20:55:08,907 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/d918056637fd414dac5930183b6b1d94 2023-04-16 20:55:09,255 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/949f83899fea4f25b934dea9e91001f5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/949f83899fea4f25b934dea9e91001f5 2023-04-16 20:55:09,260 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/A of 1e76395fa47d06dccc999b09513e8aa1 into 949f83899fea4f25b934dea9e91001f5(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-16 20:55:09,260 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:09,260 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/A, priority=13, startTime=1681678507971; duration=1sec 2023-04-16 20:55:09,261 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:09,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:09,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678569263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:09,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678569268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:09,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678569269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,317 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/ab3c443ce2a54ef68b4e0e095749959e 2023-04-16 20:55:09,321 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/78e03149066347ff9ef34f9c09f4a3ff as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/78e03149066347ff9ef34f9c09f4a3ff 2023-04-16 20:55:09,324 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/78e03149066347ff9ef34f9c09f4a3ff, entries=150, sequenceid=116, filesize=25.1 K 2023-04-16 20:55:09,325 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/d918056637fd414dac5930183b6b1d94 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d918056637fd414dac5930183b6b1d94 2023-04-16 20:55:09,329 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d918056637fd414dac5930183b6b1d94, entries=150, sequenceid=116, filesize=11.6 K 2023-04-16 20:55:09,329 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/ab3c443ce2a54ef68b4e0e095749959e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ab3c443ce2a54ef68b4e0e095749959e 2023-04-16 20:55:09,333 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ab3c443ce2a54ef68b4e0e095749959e, entries=150, sequenceid=116, filesize=11.6 K 2023-04-16 20:55:09,334 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1e76395fa47d06dccc999b09513e8aa1 in 1263ms, sequenceid=116, compaction requested=false 2023-04-16 20:55:09,334 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:09,334 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,334 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:09,334 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:09,334 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,334 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:09,334 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:09,335 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,336 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,336 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:09,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2516a757[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:09,337 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:09,337 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,341 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,341 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:09,341 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:09,341 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:09,341 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,341 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:09,342 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,342 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,342 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:09,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:09,345 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:09,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:09,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:09,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:09,345 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:09,347 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,347 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,347 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,348 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,350 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,353 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,353 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,353 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:09,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:09,353 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:09,353 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,354 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:09,354 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:09,354 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,354 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,354 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:09,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 20:55:09,398 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:09,398 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:09,398 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:09,398 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:09,398 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:09,398 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:09,413 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041676ae963757684ba4b67bb6a6257a6ec0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041676ae963757684ba4b67bb6a6257a6ec0 2023-04-16 20:55:09,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:09,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678569431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:09,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678569433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,454 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:09,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:09,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:09,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:09,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:09,457 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:09,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:09,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:09,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,459 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:09,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:09,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:09,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:09,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:09,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:09,461 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,461 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:09,462 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:09,462 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:09,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:09,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:09,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,463 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:09,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b4e9af[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:09,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:09,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,466 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:09,466 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:09,466 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:09,466 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,466 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:09,467 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:09,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:09,470 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:09,470 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:09,470 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:09,470 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:09,470 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:09,472 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,472 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,472 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,473 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,476 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,476 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,476 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:09,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:09,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:09,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,476 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,476 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,477 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:09,477 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:09,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:09,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678569538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:09,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678569538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,577 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:09,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:09,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:09,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:09,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:09,580 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:09,580 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,580 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:09,580 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:09,580 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,580 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,581 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,582 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,582 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:09,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:09,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:09,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:09,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:09,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:09,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:09,583 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,583 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:09,583 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:09,583 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:09,585 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,585 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,585 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:09,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:09,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b2ff549[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:09,589 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,589 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:09,589 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:09,589 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:09,589 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,589 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:09,590 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,590 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,590 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,590 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:09,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:09,592 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:09,592 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:09,592 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:09,592 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:09,592 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:09,593 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,593 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,594 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,595 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,596 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,598 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,598 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,598 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,598 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:09,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:09,598 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,598 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,599 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:09,599 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:09,599 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,599 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,599 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:09,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,599 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,699 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:09,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:09,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:09,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:09,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:09,702 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:09,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:09,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:09,703 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,703 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,704 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:09,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:09,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:09,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:09,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:09,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:09,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:09,709 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,709 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:09,709 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:09,709 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:09,711 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,711 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,711 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55735d4c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:09,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:09,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,714 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,714 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:09,715 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:09,715 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:09,715 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,715 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:09,717 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:09,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:09,720 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:09,720 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:09,720 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:09,720 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:09,720 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,720 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:09,721 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,721 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,721 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,723 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,726 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,726 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:09,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:09,726 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,726 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,726 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:09,726 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:09,726 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,726 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,726 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,726 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,726 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:09,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:09,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678569741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678569742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,818 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=133, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/4814e3e481f2405f8772d7796196745c 2023-04-16 20:55:09,826 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:09,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:09,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:09,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:09,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:09,830 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:09,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:09,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:09,831 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,831 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:09,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:09,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:09,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:09,833 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:09,833 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,833 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:09,833 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:09,833 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:09,833 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,834 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:09,834 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:09,834 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,834 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:09,834 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:09,834 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,834 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:09,834 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:09,836 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,837 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,837 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454f74aa[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:09,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:09,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,839 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:09,839 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:09,839 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:09,839 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,839 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:09,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:09,841 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,841 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:09,843 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:09,843 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:09,843 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:09,843 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:09,843 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:09,844 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,844 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,844 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,845 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,849 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:09,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:09,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,849 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:09,849 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:09,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,949 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:09,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:09,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:09,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:09,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:09,952 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:09,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:09,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:09,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,954 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:09,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:09,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:09,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:09,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:09,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:09,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:09,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:09,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:09,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:09,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:09,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:09,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:09,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:09,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:09,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,957 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c1043d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:09,958 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:09,958 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,959 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:09,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:09,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:09,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:09,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:09,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:09,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:09,965 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:09,965 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:09,965 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:09,965 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:09,965 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:09,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:09,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:09,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:09,970 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:09,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:09,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:09,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:09,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:09,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:09,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:09,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,970 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:09,970 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:09,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:09,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:09,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:10,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678570047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:10,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678570047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,070 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:10,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:10,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:10,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:10,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:10,073 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:10,073 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,073 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:10,073 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:10,074 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,074 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,077 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,077 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,078 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,078 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:10,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:10,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:10,079 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,079 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:10,079 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:10,079 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:10,081 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:10,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6702a91[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:10,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:10,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:10,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:10,087 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:10,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:10,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:10,089 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:10,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:10,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:10,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:10,090 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:10,091 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,091 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,091 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,091 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,096 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:10,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:10,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,096 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:10,096 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:10,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,196 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:10,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:10,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:10,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:10,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:10,199 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:10,199 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,199 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:10,199 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:10,200 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,200 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,201 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:10,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:10,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:10,202 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,202 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:10,202 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:10,202 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:10,206 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:10,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49a69e2e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:10,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:10,208 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,209 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:10,209 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,209 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:10,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:10,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,210 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:10,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:10,212 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:10,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:10,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:10,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:10,212 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:10,214 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,214 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,214 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,214 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:10,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:10,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,219 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:10,220 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:10,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,219 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,220 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,220 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,220 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,220 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:10,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/033682834de44970b6ef6b137d0f0e03 2023-04-16 20:55:10,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:10,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678570266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:10,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678570274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:10,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678570282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,320 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:10,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:10,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:10,323 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:10,323 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,323 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:10,323 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:10,323 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,323 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,324 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,325 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,325 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:10,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:10,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:10,326 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,326 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:10,326 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:10,326 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:10,328 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:10,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f24994[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:10,331 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:10,331 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,332 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,332 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:10,332 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:10,335 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:10,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:10,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:10,337 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:10,337 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:10,337 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:10,337 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:10,337 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:10,338 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,338 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,338 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,338 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,340 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:10,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:10,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:10,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,344 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:10,344 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:10,444 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:10,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:10,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:10,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:10,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:10,447 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:10,447 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,447 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:10,447 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:10,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,449 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,449 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,449 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:10,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:10,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:10,451 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,451 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:10,451 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:10,451 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:10,455 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:10,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4da84bbb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:10,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:10,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,458 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:10,459 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,459 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:10,460 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:10,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:10,463 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:10,463 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:10,463 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:10,463 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:10,463 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:10,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,466 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,467 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,469 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:10,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:10,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,470 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,470 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,470 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,470 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,470 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:10,470 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:10,469 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:10,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:10,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678570553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:10,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678570554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,570 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:10,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:10,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:10,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:10,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:10,573 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:10,573 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,573 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:10,573 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:10,573 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,574 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,575 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:10,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:10,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:10,576 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,577 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:10,577 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:10,577 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:10,578 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,578 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:10,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a8bd43d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:10,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:10,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,583 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:10,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:10,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:10,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:10,584 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:10,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:10,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:10,586 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:10,586 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:10,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:10,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:10,587 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:10,588 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,588 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:10,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,588 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,589 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,590 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,592 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:10,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:10,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:10,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:10,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,592 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:10,592 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:10,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:10,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:10,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/1447779e86c143aa9f21e41f1b432fb2 2023-04-16 20:55:10,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/4814e3e481f2405f8772d7796196745c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4814e3e481f2405f8772d7796196745c 2023-04-16 20:55:10,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4814e3e481f2405f8772d7796196745c, entries=200, sequenceid=133, filesize=32.0 K 2023-04-16 20:55:10,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/033682834de44970b6ef6b137d0f0e03 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/033682834de44970b6ef6b137d0f0e03 2023-04-16 20:55:10,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/033682834de44970b6ef6b137d0f0e03, entries=150, sequenceid=133, filesize=11.7 K 2023-04-16 20:55:10,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/1447779e86c143aa9f21e41f1b432fb2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/1447779e86c143aa9f21e41f1b432fb2 2023-04-16 20:55:10,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/1447779e86c143aa9f21e41f1b432fb2, entries=150, sequenceid=133, filesize=11.7 K 2023-04-16 20:55:10,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1e76395fa47d06dccc999b09513e8aa1 in 1256ms, sequenceid=133, compaction requested=true 2023-04-16 20:55:10,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:10,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:10,654 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:10,654 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:10,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:10,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:10,655 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84434 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:10,655 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:10,655 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/A is initiating minor compaction (all files) 2023-04-16 20:55:10,655 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/B is initiating minor compaction (all files) 2023-04-16 20:55:10,655 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/A in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,655 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/B in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,655 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/949f83899fea4f25b934dea9e91001f5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/78e03149066347ff9ef34f9c09f4a3ff, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4814e3e481f2405f8772d7796196745c] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=82.5 K 2023-04-16 20:55:10,655 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/e7814a62d9ee43a38251dff6fc466d56, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d918056637fd414dac5930183b6b1d94, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/033682834de44970b6ef6b137d0f0e03] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=35.1 K 2023-04-16 20:55:10,656 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 949f83899fea4f25b934dea9e91001f5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678505242 2023-04-16 20:55:10,656 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e7814a62d9ee43a38251dff6fc466d56, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678505242 2023-04-16 20:55:10,656 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 78e03149066347ff9ef34f9c09f4a3ff, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681678507497 2023-04-16 20:55:10,656 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d918056637fd414dac5930183b6b1d94, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681678507497 2023-04-16 20:55:10,656 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 4814e3e481f2405f8772d7796196745c, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681678508131 2023-04-16 20:55:10,656 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 033682834de44970b6ef6b137d0f0e03, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681678508140 2023-04-16 20:55:10,664 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#B#compaction#356 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:10,670 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#A#compaction#357 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:10,672 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/876801c2fd85469ca7e82f53b54f71a4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/876801c2fd85469ca7e82f53b54f71a4 2023-04-16 20:55:10,676 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/B of 1e76395fa47d06dccc999b09513e8aa1 into 876801c2fd85469ca7e82f53b54f71a4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:10,676 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:10,676 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/B, priority=13, startTime=1681678510654; duration=0sec 2023-04-16 20:55:10,676 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:10,676 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:10,677 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:10,677 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/C is initiating minor compaction (all files) 2023-04-16 20:55:10,677 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/C in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,677 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c16740a9b5104898a6595c105d17585a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ab3c443ce2a54ef68b4e0e095749959e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/1447779e86c143aa9f21e41f1b432fb2] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=35.1 K 2023-04-16 20:55:10,678 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c16740a9b5104898a6595c105d17585a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678505242 2023-04-16 20:55:10,679 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ab3c443ce2a54ef68b4e0e095749959e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681678507497 2023-04-16 20:55:10,679 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1447779e86c143aa9f21e41f1b432fb2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681678508140 2023-04-16 20:55:10,693 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:10,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:10,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:10,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:10,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:10,696 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:10,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:10,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:10,697 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,697 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#C#compaction#358 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:10,697 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,701 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:10,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:10,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:10,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:10,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:10,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:10,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:10,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:10,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:10,704 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:10,704 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:10,704 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-16 20:55:10,704 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:10,705 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:10,705 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:10,705 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:10,705 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:10,705 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:11,110 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/029469ca8f24496890184bd68de492cf as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/029469ca8f24496890184bd68de492cf 2023-04-16 20:55:11,113 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/c687c51caaa84e7e9aeaa85cdb18d060 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/c687c51caaa84e7e9aeaa85cdb18d060 2023-04-16 20:55:11,114 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/C of 1e76395fa47d06dccc999b09513e8aa1 into 029469ca8f24496890184bd68de492cf(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:11,114 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:11,114 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/C, priority=13, startTime=1681678510654; duration=0sec 2023-04-16 20:55:11,115 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:11,117 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/A of 1e76395fa47d06dccc999b09513e8aa1 into c687c51caaa84e7e9aeaa85cdb18d060(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:11,117 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:11,117 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/A, priority=13, startTime=1681678510654; duration=0sec 2023-04-16 20:55:11,117 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:11,130 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041611221dc25f8b452d9f7366e0037b3666 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041611221dc25f8b452d9f7366e0037b3666 2023-04-16 20:55:11,135 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/435b2be23f6344d88e9403488e49ab96 2023-04-16 20:55:11,144 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/7120f8842db04ee8a1c35b1c782cf81b 2023-04-16 20:55:11,170 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/7d983354f2d54001bc59f5e18b111c3a 2023-04-16 20:55:11,178 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/435b2be23f6344d88e9403488e49ab96 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/435b2be23f6344d88e9403488e49ab96 2023-04-16 20:55:11,183 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/435b2be23f6344d88e9403488e49ab96, entries=150, sequenceid=152, filesize=25.3 K 2023-04-16 20:55:11,184 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/7120f8842db04ee8a1c35b1c782cf81b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/7120f8842db04ee8a1c35b1c782cf81b 2023-04-16 20:55:11,187 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/7120f8842db04ee8a1c35b1c782cf81b, entries=150, sequenceid=152, filesize=11.8 K 2023-04-16 20:55:11,187 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/7d983354f2d54001bc59f5e18b111c3a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7d983354f2d54001bc59f5e18b111c3a 2023-04-16 20:55:11,193 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7d983354f2d54001bc59f5e18b111c3a, entries=150, sequenceid=152, filesize=11.8 K 2023-04-16 20:55:11,194 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 1e76395fa47d06dccc999b09513e8aa1 in 490ms, sequenceid=152, compaction requested=false 2023-04-16 20:55:11,194 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:11,194 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,194 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:11,194 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:11,194 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,194 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:11,194 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:11,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:11,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@372174ac[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:11,198 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:11,198 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,199 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,199 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:11,199 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:11,199 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:11,199 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,200 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,201 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:11,204 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:11,204 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:11,204 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:11,204 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:11,204 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:11,206 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,206 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,206 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,208 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,211 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,211 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,212 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,212 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,212 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,212 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:11,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:11,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,212 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,212 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,212 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,212 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:11,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:11,212 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:11,312 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:11,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:11,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:11,316 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:11,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:11,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:11,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,317 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,318 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,318 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:11,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:11,320 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:11,320 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,320 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:11,320 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:11,320 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,320 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:11,321 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:11,321 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,321 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:11,321 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:11,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,322 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,322 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,322 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:11,324 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:11,324 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@527a2535[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:11,326 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:11,326 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,326 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:11,330 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,331 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:11,331 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,331 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:11,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:11,333 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:11,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:11,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:11,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:11,333 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:11,335 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,335 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,335 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,337 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:11,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:11,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:11,340 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,341 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,341 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:11,341 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:11,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,341 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,341 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,341 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,441 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:11,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:11,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:11,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:11,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:11,444 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:11,444 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:11,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:11,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,448 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,448 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:11,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:11,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:11,450 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,450 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:11,450 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:11,450 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:11,451 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,451 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d005952[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:11,453 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:11,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,454 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:11,455 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,455 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:11,456 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:11,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:11,458 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:11,459 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:11,459 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:11,459 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:11,459 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:11,461 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,461 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,461 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,462 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,463 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:11,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:11,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,466 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:11,466 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:11,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,566 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:11,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:11,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:11,570 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:11,570 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,570 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:11,570 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:11,571 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,571 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,572 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,572 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,572 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,572 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:11,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:11,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:11,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:11,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:11,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:11,580 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,580 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:11,580 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:11,580 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:11,582 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,582 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,582 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,582 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:11,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49be44f3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:11,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:11,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,587 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:11,587 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:11,587 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:11,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:11,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:11,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,588 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:11,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:11,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:11,591 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:11,591 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:11,591 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:11,591 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:11,591 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:11,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,593 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,594 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,597 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:11,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:11,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,598 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,598 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:11,598 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:11,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:11,698 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:11,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:11,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:11,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:11,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:11,702 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:11,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:11,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:11,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,704 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,705 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,705 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:11,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:11,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:11,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:11,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:11,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:11,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:11,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:11,706 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,706 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:11,707 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 20:55:11,707 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:11,707 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:11,707 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:11,707 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:11,707 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:11,707 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:11,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:11,741 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:11,744 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304165ae48715c63a4362b5bf5f4c2903eb97 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165ae48715c63a4362b5bf5f4c2903eb97 2023-04-16 20:55:11,751 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=164, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/a8fa71eab8d34baa890da2f0443bdef8 2023-04-16 20:55:11,779 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/3d10bab05d0b45bfb155ff04d178c8f9 2023-04-16 20:55:11,798 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/2acc2b515aca48b9b20222a56e977afb 2023-04-16 20:55:11,803 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/a8fa71eab8d34baa890da2f0443bdef8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/a8fa71eab8d34baa890da2f0443bdef8 2023-04-16 20:55:11,806 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/a8fa71eab8d34baa890da2f0443bdef8, entries=150, sequenceid=164, filesize=25.3 K 2023-04-16 20:55:11,807 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/3d10bab05d0b45bfb155ff04d178c8f9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3d10bab05d0b45bfb155ff04d178c8f9 2023-04-16 20:55:11,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678511777, max timestamp=1681678511810Num uniques 859; 2023-04-16 20:55:11,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:11,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678511777, max timestamp=1681678511810Num uniques 859; 2023-04-16 20:55:11,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:11,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678511777, max timestamp=1681678511810Num uniques 859; 2023-04-16 20:55:11,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:11,813 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3d10bab05d0b45bfb155ff04d178c8f9, entries=100, sequenceid=164, filesize=9.4 K 2023-04-16 20:55:11,814 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/2acc2b515aca48b9b20222a56e977afb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/2acc2b515aca48b9b20222a56e977afb 2023-04-16 20:55:11,819 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/2acc2b515aca48b9b20222a56e977afb, entries=100, sequenceid=164, filesize=9.4 K 2023-04-16 20:55:11,820 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=40.25 KB/41220 for 1e76395fa47d06dccc999b09513e8aa1 in 114ms, sequenceid=164, compaction requested=true 2023-04-16 20:55:11,820 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:11,820 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:11,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:11,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:11,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:11,822 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,822 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:11,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7506cd7c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:11,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:11,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:11,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:11,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:11,826 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:11,826 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:11,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-16 20:55:11,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:11,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:11,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:11,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:11,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:11,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:11,827 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:11,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:11,836 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:11,836 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:11,836 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:11,836 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:11,836 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:11,843 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,843 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,843 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,844 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,845 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,849 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:11,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:11,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,849 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,850 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,850 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,850 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,850 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:11,850 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:11,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:11,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,858 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304167d19dcdcc94649bc95640cacc8923438 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167d19dcdcc94649bc95640cacc8923438 2023-04-16 20:55:11,875 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=189, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/87b3edcdb5cf4942a65a10597c0f0437 2023-04-16 20:55:11,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678511902, max timestamp=1681678511926Num uniques 859; 2023-04-16 20:55:11,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:11,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678511902, max timestamp=1681678511926Num uniques 859; 2023-04-16 20:55:11,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:11,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678511902, max timestamp=1681678511926Num uniques 859; 2023-04-16 20:55:11,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:11,950 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:11,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:11,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:11,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:11,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:11,953 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:11,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:11,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:11,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,955 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,956 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:11,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:11,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:11,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:11,957 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,957 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:11,957 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:11,957 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:11,960 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:11,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76560d64[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:11,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:11,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:11,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:11,963 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:11,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:11,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:11,967 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:11,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:11,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:11,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:11,967 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:11,968 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,968 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,968 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,969 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:11,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,976 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:11,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:11,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,978 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:11,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:11,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:11,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:11,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:11,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:11,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:11,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:11,979 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:11,979 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:11,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678572008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678572008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,079 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:12,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:12,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:12,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:12,083 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:12,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:12,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:12,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,084 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:12,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:12,086 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,086 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:12,086 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:12,086 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:12,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:12,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:12,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29c31035[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:12,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:12,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,093 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,093 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:12,093 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:12,093 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:12,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:12,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:12,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:12,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:12,098 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:12,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:12,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:12,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:12,098 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:12,100 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,100 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,100 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,100 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,101 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,103 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,103 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,103 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,103 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,103 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,103 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:12,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:12,103 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,104 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:12,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,104 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,104 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:12,104 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:12,104 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678572116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678572121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,204 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:12,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:12,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:12,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:12,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:12,207 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:12,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:12,207 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:12,208 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,208 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,209 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,209 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,209 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,209 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:12,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:12,211 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:12,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:12,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:12,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12614022[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:12,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:12,215 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,216 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,216 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:12,216 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:12,216 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:12,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:12,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:12,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:12,220 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:12,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:12,221 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:12,221 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:12,221 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:12,222 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,223 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,223 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,224 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,229 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:12,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:12,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,230 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:12,230 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:12,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:12,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678572285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,292 DEBUG [Thread-2477] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:55:12,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678572290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,296 DEBUG [Thread-2471] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:55:12,300 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/9411d04887a34819a85aa257082db4e5 2023-04-16 20:55:12,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678572296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,302 DEBUG [Thread-2473] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:55:12,330 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:12,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:12,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:12,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678572324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678572324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:12,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:12,333 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:12,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:12,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:12,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,335 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:12,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:12,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:12,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:12,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:12,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:12,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:12,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:12,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:12,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:12,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:12,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:12,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:12,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:12,339 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,340 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,340 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:12,342 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:12,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51cf123f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:12,342 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:12,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:12,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:12,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:12,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:12,347 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:12,348 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:12,348 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:12,348 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:12,348 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:12,349 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,349 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,349 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,350 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,351 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,354 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,354 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,354 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:12,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:12,354 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:12,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,355 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:12,355 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:12,354 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,355 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,355 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,355 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,355 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,355 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,455 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:12,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:12,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:12,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:12,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:12,458 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:12,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:12,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:12,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,459 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,460 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:12,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:12,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:12,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:12,462 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,462 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:12,462 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:12,462 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:12,464 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,464 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,464 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,464 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:12,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:12,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17b4084f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:12,466 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,466 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:12,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:12,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:12,467 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,467 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:12,468 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,468 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:12,468 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,468 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:12,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:12,470 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:12,470 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:12,470 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:12,470 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:12,470 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:12,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,472 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,475 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,477 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:12,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:12,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,478 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:12,478 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:12,478 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,478 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,478 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,478 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,478 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,578 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:12,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:12,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:12,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:12,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:12,581 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:12,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:12,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:12,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,581 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,582 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,582 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,582 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,582 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:12,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:12,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:12,584 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,584 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:12,584 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:12,584 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:12,585 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,585 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:12,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2647f034[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:12,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:12,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,588 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:12,588 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,588 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:12,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:12,588 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:12,591 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,591 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,591 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,591 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:12,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:12,593 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:12,593 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:12,593 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:12,593 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:12,593 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:12,594 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,594 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,594 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,595 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,596 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,600 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,600 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,600 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:12,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:12,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,600 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,600 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,600 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:12,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:12,600 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,600 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:12,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,601 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678572634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:12,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678572635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,701 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:12,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:12,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:12,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:12,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:12,704 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:12,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:12,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:12,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,704 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,705 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,705 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,706 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:12,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:12,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:12,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:12,707 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:12,707 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,707 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:12,707 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:12,707 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:12,707 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,707 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:12,708 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:12,708 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:12,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,709 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:12,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ebfd48e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:12,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:12,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,712 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:12,712 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:12,712 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:12,712 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,712 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:12,713 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/f4c4384fddb642f7a0ef08f0ea581eeb 2023-04-16 20:55:12,714 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,714 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:12,714 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,714 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:12,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/87b3edcdb5cf4942a65a10597c0f0437 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/87b3edcdb5cf4942a65a10597c0f0437 2023-04-16 20:55:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:12,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:12,717 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:12,717 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:12,717 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:12,718 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:12,718 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:12,719 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,719 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,719 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:12,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:12,720 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/87b3edcdb5cf4942a65a10597c0f0437, entries=250, sequenceid=189, filesize=38.9 K 2023-04-16 20:55:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:12,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/9411d04887a34819a85aa257082db4e5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9411d04887a34819a85aa257082db4e5 2023-04-16 20:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9411d04887a34819a85aa257082db4e5, entries=150, sequenceid=189, filesize=11.8 K 2023-04-16 20:55:12,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:12,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:12,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:12,725 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:12,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:12,725 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,725 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:12,725 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:12,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/f4c4384fddb642f7a0ef08f0ea581eeb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f4c4384fddb642f7a0ef08f0ea581eeb 2023-04-16 20:55:12,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:12,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f4c4384fddb642f7a0ef08f0ea581eeb, entries=150, sequenceid=189, filesize=11.8 K 2023-04-16 20:55:12,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=134.18 KB/137400 for 1e76395fa47d06dccc999b09513e8aa1 in 903ms, sequenceid=189, compaction requested=true 2023-04-16 20:55:12,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:12,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:12,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:12,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:12,730 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:12,730 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:12,731 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45958 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:12,731 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117772 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:12,731 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/B is initiating minor compaction (all files) 2023-04-16 20:55:12,731 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/A is initiating minor compaction (all files) 2023-04-16 20:55:12,731 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/B in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,731 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/A in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,731 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/876801c2fd85469ca7e82f53b54f71a4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/7120f8842db04ee8a1c35b1c782cf81b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3d10bab05d0b45bfb155ff04d178c8f9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9411d04887a34819a85aa257082db4e5] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=44.9 K 2023-04-16 20:55:12,731 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/c687c51caaa84e7e9aeaa85cdb18d060, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/435b2be23f6344d88e9403488e49ab96, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/a8fa71eab8d34baa890da2f0443bdef8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/87b3edcdb5cf4942a65a10597c0f0437] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=115.0 K 2023-04-16 20:55:12,732 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 876801c2fd85469ca7e82f53b54f71a4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681678508140 2023-04-16 20:55:12,732 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c687c51caaa84e7e9aeaa85cdb18d060, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681678508140 2023-04-16 20:55:12,732 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7120f8842db04ee8a1c35b1c782cf81b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681678509424 2023-04-16 20:55:12,732 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 435b2be23f6344d88e9403488e49ab96, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681678509424 2023-04-16 20:55:12,732 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3d10bab05d0b45bfb155ff04d178c8f9, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681678511620 2023-04-16 20:55:12,733 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting a8fa71eab8d34baa890da2f0443bdef8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681678511620 2023-04-16 20:55:12,733 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9411d04887a34819a85aa257082db4e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1681678511815 2023-04-16 20:55:12,733 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 87b3edcdb5cf4942a65a10597c0f0437, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1681678511810 2023-04-16 20:55:12,757 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#B#compaction#369 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:12,759 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#A#compaction#368 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:12,794 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/1522526975094c1ab9d59598109a5877 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/1522526975094c1ab9d59598109a5877 2023-04-16 20:55:12,797 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/A of 1e76395fa47d06dccc999b09513e8aa1 into 1522526975094c1ab9d59598109a5877(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:12,797 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:12,797 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/A, priority=12, startTime=1681678512730; duration=0sec 2023-04-16 20:55:12,798 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:12,798 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:12,799 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45958 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:12,799 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/C is initiating minor compaction (all files) 2023-04-16 20:55:12,799 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/C in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,799 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/029469ca8f24496890184bd68de492cf, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7d983354f2d54001bc59f5e18b111c3a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/2acc2b515aca48b9b20222a56e977afb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f4c4384fddb642f7a0ef08f0ea581eeb] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=44.9 K 2023-04-16 20:55:12,799 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 029469ca8f24496890184bd68de492cf, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1681678508140 2023-04-16 20:55:12,800 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7d983354f2d54001bc59f5e18b111c3a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681678509424 2023-04-16 20:55:12,800 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2acc2b515aca48b9b20222a56e977afb, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681678511620 2023-04-16 20:55:12,800 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting f4c4384fddb642f7a0ef08f0ea581eeb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1681678511815 2023-04-16 20:55:12,806 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#C#compaction#370 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:12,814 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/e8a3ea3d0a2e48c0b991f56cf398fe33 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/e8a3ea3d0a2e48c0b991f56cf398fe33 2023-04-16 20:55:12,818 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/C of 1e76395fa47d06dccc999b09513e8aa1 into e8a3ea3d0a2e48c0b991f56cf398fe33(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:12,818 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:12,818 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/C, priority=12, startTime=1681678512730; duration=0sec 2023-04-16 20:55:12,818 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:12,825 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:12,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:12,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:12,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:12,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:12,828 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:12,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,829 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:12,829 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:12,829 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,829 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,830 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:12,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:12,831 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,831 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:12,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:12,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:12,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:12,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:12,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:12,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:12,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:12,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:12,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:12,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:12,832 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:12,832 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:12,832 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 20:55:12,833 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:12,833 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:12,833 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:12,833 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:12,833 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:12,833 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:13,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:13,146 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:13,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:13,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678573169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:13,175 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/8162bf7f2b9e489c80dbaf7bda4897f9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/8162bf7f2b9e489c80dbaf7bda4897f9 2023-04-16 20:55:13,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:13,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678573172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:13,178 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/B of 1e76395fa47d06dccc999b09513e8aa1 into 8162bf7f2b9e489c80dbaf7bda4897f9(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:13,178 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:13,178 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/B, priority=12, startTime=1681678512730; duration=0sec 2023-04-16 20:55:13,178 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:13,244 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416ca85afb946b144cebc63d3ab2a32ce9f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ca85afb946b144cebc63d3ab2a32ce9f 2023-04-16 20:55:13,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:13,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678573273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:13,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:13,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678573276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:13,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:13,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678573477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:13,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:13,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678573480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:13,650 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=228, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/66380952164b4beda5f6b82712084bd6 2023-04-16 20:55:13,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:13,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678573781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:13,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:13,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678573788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,061 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/0a9462116cd54a8cb4718e049c43c855 2023-04-16 20:55:14,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:14,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46756 deadline: 1681678574290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:14,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46736 deadline: 1681678574295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,475 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/93a7813e34e24fb2af1264555be13b5f 2023-04-16 20:55:14,479 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/66380952164b4beda5f6b82712084bd6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/66380952164b4beda5f6b82712084bd6 2023-04-16 20:55:14,482 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/66380952164b4beda5f6b82712084bd6, entries=150, sequenceid=228, filesize=25.3 K 2023-04-16 20:55:14,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-16 20:55:14,483 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/0a9462116cd54a8cb4718e049c43c855 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/0a9462116cd54a8cb4718e049c43c855 2023-04-16 20:55:14,486 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/0a9462116cd54a8cb4718e049c43c855, entries=150, sequenceid=228, filesize=11.8 K 2023-04-16 20:55:14,487 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/93a7813e34e24fb2af1264555be13b5f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/93a7813e34e24fb2af1264555be13b5f 2023-04-16 20:55:14,490 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/93a7813e34e24fb2af1264555be13b5f, entries=150, sequenceid=228, filesize=11.8 K 2023-04-16 20:55:14,491 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 1e76395fa47d06dccc999b09513e8aa1 in 1659ms, sequenceid=228, compaction requested=false 2023-04-16 20:55:14,491 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:14,491 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:14,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:14,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:14,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:14,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:14,500 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:14,500 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:14,500 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:14,500 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:14,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:14,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c8678c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:14,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:14,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:14,503 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:14,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:14,503 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:14,503 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:14,503 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:14,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:14,504 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:14,504 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,504 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:14,504 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:14,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:14,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:14,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:14,507 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:14,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:14,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:14,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:14,507 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:14,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:14,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:14,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:14,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:14,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:14,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:14,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:14,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:14,510 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:14,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:14,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:14,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:14,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:14,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:14,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:14,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:14,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:14,516 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:14,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,516 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:14,516 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:14,516 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:14,516 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:14,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:14,616 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:14,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:14,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:14,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:14,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:14,620 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:14,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:14,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:14,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:14,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:14,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:14,621 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,621 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:14,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:14,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:14,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:14,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:14,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:14,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:14,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:14,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:14,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:14,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:14,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:14,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:14,623 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:14,623 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:14,624 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:55:14,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:14,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:14,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:14,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:14,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:14,624 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:14,635 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041641cf771dea1844c592d0e8a5a6ece646 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041641cf771dea1844c592d0e8a5a6ece646 2023-04-16 20:55:15,040 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/3123fc812dbf4997b4022bdfb6e5fd50 2023-04-16 20:55:15,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:15,299 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:15,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678515328, max timestamp=1681678515340Num uniques 859; 2023-04-16 20:55:15,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:15,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678515328, max timestamp=1681678515340Num uniques 859; 2023-04-16 20:55:15,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:15,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678515328, max timestamp=1681678515340Num uniques 859; 2023-04-16 20:55:15,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:15,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:15,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678575392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:15,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678575400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,449 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/1be54ca1494649a8bf28aad47e8df9c7 2023-04-16 20:55:15,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:15,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678575503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:15,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678575507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:15,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678575707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:15,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678575714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,859 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/edfb46f78e6b49a9a27d09c5484b0b69 2023-04-16 20:55:15,862 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/3123fc812dbf4997b4022bdfb6e5fd50 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/3123fc812dbf4997b4022bdfb6e5fd50 2023-04-16 20:55:15,866 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/3123fc812dbf4997b4022bdfb6e5fd50, entries=150, sequenceid=239, filesize=25.3 K 2023-04-16 20:55:15,866 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/1be54ca1494649a8bf28aad47e8df9c7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1be54ca1494649a8bf28aad47e8df9c7 2023-04-16 20:55:15,870 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1be54ca1494649a8bf28aad47e8df9c7, entries=150, sequenceid=239, filesize=11.8 K 2023-04-16 20:55:15,870 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/edfb46f78e6b49a9a27d09c5484b0b69 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/edfb46f78e6b49a9a27d09c5484b0b69 2023-04-16 20:55:15,873 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/edfb46f78e6b49a9a27d09c5484b0b69, entries=150, sequenceid=239, filesize=11.8 K 2023-04-16 20:55:15,874 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1e76395fa47d06dccc999b09513e8aa1 in 1251ms, sequenceid=239, compaction requested=true 2023-04-16 20:55:15,874 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:15,874 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:15,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:15,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:15,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:15,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:15,876 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:15,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:15,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:15,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:15,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2089e34a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:15,877 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:15,877 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:15,878 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:15,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:15,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:15,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:15,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:15,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:15,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:15,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,881 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:15,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:15,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:15,881 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:15,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:15,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:15,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:15,881 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:15,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:15,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:15,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:15,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:15,884 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:15,885 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:15,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:15,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:15,887 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:15,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:15,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:15,887 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:15,887 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:15,887 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:15,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:15,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:15,988 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:15,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:15,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:15,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:15,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:15,992 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:15,992 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:15,992 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:15,992 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:15,993 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:15,993 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:15,994 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,994 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:15,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:15,995 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:15,995 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:15,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:15,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:15,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:15,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:15,996 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:15,996 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:15,996 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:15,996 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:15,996 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:15,996 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:15,996 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:15,996 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-16 20:55:15,996 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:15,996 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:15,997 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:15,997 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:15,997 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:15,997 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:16,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:16,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:16,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678576033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678576035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678576138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678576141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46798 deadline: 1681678576293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,298 DEBUG [Thread-2477] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:55:16,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46744 deadline: 1681678576317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,322 DEBUG [Thread-2471] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:55:16,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46770 deadline: 1681678576339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,344 DEBUG [Thread-2473] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8201 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:55:16,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678576342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678576350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,409 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304161b5b7f6b9cac4ffebdac9bf46eebfea6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161b5b7f6b9cac4ffebdac9bf46eebfea6 2023-04-16 20:55:16,414 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/2d999dff8cc944d890beb92cde812321 2023-04-16 20:55:16,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678576653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:16,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678576658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,823 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/6709dd3730ff4eb784b200f5649f85cc 2023-04-16 20:55:16,833 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/fdda4a882df146f1acad01498a828d4b 2023-04-16 20:55:16,846 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/2d999dff8cc944d890beb92cde812321 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/2d999dff8cc944d890beb92cde812321 2023-04-16 20:55:16,851 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/2d999dff8cc944d890beb92cde812321, entries=150, sequenceid=275, filesize=25.4 K 2023-04-16 20:55:16,851 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/6709dd3730ff4eb784b200f5649f85cc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/6709dd3730ff4eb784b200f5649f85cc 2023-04-16 20:55:16,855 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/6709dd3730ff4eb784b200f5649f85cc, entries=150, sequenceid=275, filesize=11.9 K 2023-04-16 20:55:16,856 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/fdda4a882df146f1acad01498a828d4b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/fdda4a882df146f1acad01498a828d4b 2023-04-16 20:55:16,859 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/fdda4a882df146f1acad01498a828d4b, entries=150, sequenceid=275, filesize=11.9 K 2023-04-16 20:55:16,860 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 1e76395fa47d06dccc999b09513e8aa1 in 864ms, sequenceid=275, compaction requested=true 2023-04-16 20:55:16,860 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:16,860 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:16,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:16,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:16,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,864 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:16,864 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:16,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,870 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:16,870 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:16,870 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:16,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:16,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d6613f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:16,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:16,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:16,873 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:16,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:16,874 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,874 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:16,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:16,875 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:16,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:16,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:16,878 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:16,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:16,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:16,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:16,878 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:16,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:16,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:16,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:16,880 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:16,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:16,880 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:16,882 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:16,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,885 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:16,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:16,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:16,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:16,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:16,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:16,885 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:16,886 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:16,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:16,885 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:16,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:16,886 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,886 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:16,886 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:16,886 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:16,986 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:16,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:16,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:16,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:16,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:16,991 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:16,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:16,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:16,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:16,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:16,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:16,992 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:16,992 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:16,993 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:16,993 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:16,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:16,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:17,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:17,001 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:17,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:17,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:17,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:17,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:17,018 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:55:17,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:17,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:17,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:17,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:17,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:17,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:17,034 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416b82f680e96d04aeab74571ee3209cbbf to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b82f680e96d04aeab74571ee3209cbbf 2023-04-16 20:55:17,057 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=285, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/8d27093da8884c8c906fd7b9bd411320 2023-04-16 20:55:17,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:17,183 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:17,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681678517250, max timestamp=1681678517264Num uniques 859; 2023-04-16 20:55:17,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:17,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:17,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678577265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681678517250, max timestamp=1681678517264Num uniques 859; 2023-04-16 20:55:17,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:17,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681678517250, max timestamp=1681678517264Num uniques 859; 2023-04-16 20:55:17,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:17,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:17,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678577350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:17,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678577367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:17,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678577455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,486 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/d4723fc16025444cadb0715119be957b 2023-04-16 20:55:17,496 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/9bf2cf418fcf443e9326b18d59bee61e 2023-04-16 20:55:17,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/8d27093da8884c8c906fd7b9bd411320 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/8d27093da8884c8c906fd7b9bd411320 2023-04-16 20:55:17,514 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/8d27093da8884c8c906fd7b9bd411320, entries=100, sequenceid=285, filesize=18.6 K 2023-04-16 20:55:17,515 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/d4723fc16025444cadb0715119be957b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d4723fc16025444cadb0715119be957b 2023-04-16 20:55:17,519 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d4723fc16025444cadb0715119be957b, entries=100, sequenceid=285, filesize=9.5 K 2023-04-16 20:55:17,519 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/9bf2cf418fcf443e9326b18d59bee61e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/9bf2cf418fcf443e9326b18d59bee61e 2023-04-16 20:55:17,524 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/9bf2cf418fcf443e9326b18d59bee61e, entries=100, sequenceid=285, filesize=9.5 K 2023-04-16 20:55:17,525 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1e76395fa47d06dccc999b09513e8aa1 in 507ms, sequenceid=285, compaction requested=true 2023-04-16 20:55:17,525 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:17,525 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:17,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,527 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,527 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,528 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:17,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790dd92f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:17,531 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:17,531 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,532 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:17,533 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,533 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:17,534 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,534 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:17,535 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,535 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:17,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:17,537 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:17,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:17,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:17,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:17,537 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:17,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:17,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,540 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,541 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,542 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,544 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:17,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:17,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:17,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:17,545 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:17,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:17,545 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:17,544 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,545 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:17,545 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,545 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,545 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,545 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:17,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:17,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-16 20:55:17,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:17,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:17,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:17,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:17,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:17,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:17,609 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304165cd131a7b6354160b03c074418873e6b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165cd131a7b6354160b03c074418873e6b 2023-04-16 20:55:17,621 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/25b21c6e1bcd4d039e76fca6813a51ac 2023-04-16 20:55:17,645 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:17,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:17,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:17,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:17,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:17,654 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,656 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:17,656 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:17,656 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,656 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:17,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:17,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:17,657 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:17,657 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:17,657 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:17,657 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:17,658 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:17,658 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:17,658 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:17,658 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:17,660 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:17,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dad3585[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:17,665 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:17,665 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,666 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:17,666 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:17,666 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:17,666 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:17,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:17,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:17,668 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:17,668 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:17,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:17,671 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:17,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:17,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:17,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:17,671 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:17,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:17,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678577660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:17,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678577661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,673 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,673 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:17,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,673 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:17,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,675 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,679 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:17,679 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:17,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:17,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,680 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:17,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:17,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:17,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,680 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:17,680 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:17,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:17,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:17,780 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:17,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:17,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:17,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:17,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678577773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:17,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:17,785 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:17,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:17,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:17,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:17,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,787 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:17,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:17,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:17,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:17,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:17,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:17,789 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:17,789 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:17,789 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:17,789 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,790 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,791 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,791 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:17,792 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:17,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42d100c9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:17,792 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,793 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,793 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,793 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:17,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:17,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:17,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:17,799 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:17,799 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,799 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,799 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:17,801 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:17,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:17,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:17,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:17,801 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:17,802 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,802 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,803 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:17,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,805 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:17,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:17,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:17,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,809 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:17,809 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:17,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:17,809 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,809 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,809 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,809 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:17,809 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:17,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:17,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:17,909 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:17,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:17,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:17,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:17,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:17,913 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:17,913 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:17,913 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:17,913 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:17,914 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,914 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,915 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:17,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:17,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:17,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:17,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:17,917 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:17,917 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:17,917 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:17,917 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:17,920 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:17,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48844e35[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:17,922 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:17,922 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,923 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:17,923 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,923 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:17,924 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,924 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:17,924 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,924 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:17,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:17,926 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:17,926 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:17,926 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:17,926 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:17,926 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:17,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:17,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,928 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:17,929 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:17,934 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:17,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:17,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:17,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:17,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:17,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:17,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:17,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:17,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:17,934 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:17,934 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:17,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:17,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678577974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:17,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:17,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678577985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/9c95063bb6484ce79104e2dbab885dd4 2023-04-16 20:55:18,034 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:18,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:18,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:18,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:18,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:18,039 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:18,039 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,039 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:18,039 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:18,040 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,040 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,042 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,042 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,042 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,042 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:18,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:18,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:18,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:18,044 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:18,044 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,044 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:18,048 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,048 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:18,048 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:18,048 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,052 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:18,052 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:18,053 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,053 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,053 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,054 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd732ab[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:18,055 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:18,055 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,056 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:18,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:18,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,058 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:18,058 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,058 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:18,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:18,061 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:18,061 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:18,061 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:18,061 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:18,062 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:18,064 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,064 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,064 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,065 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,068 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,070 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:18,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:18,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:18,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,070 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:18,070 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:18,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,171 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:18,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:18,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:18,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:18,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:18,174 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:18,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:18,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:18,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,174 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,175 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:18,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:18,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:18,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:18,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:18,177 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,177 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:18,177 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:18,177 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:18,179 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,179 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,180 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,180 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b1225c9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:18,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:18,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,183 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,183 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:18,183 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,183 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:18,183 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:18,183 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:18,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:18,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,184 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,184 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:18,186 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:18,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:18,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:18,186 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:18,186 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:18,191 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,191 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,191 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,192 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,194 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:18,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:18,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:18,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,196 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:18,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,196 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,196 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:18,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,196 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,296 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:18,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:18,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678578292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:18,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:18,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:18,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:18,304 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:18,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:18,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:18,305 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,305 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,306 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,307 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,308 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,308 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:18,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:18,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:18,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:18,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:18,310 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:18,312 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,313 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,313 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@510fa92a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:18,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:18,316 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,322 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,322 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:18,322 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:18,322 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:18,322 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,322 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:18,323 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:18,326 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:18,326 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:18,326 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:18,326 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:18,326 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:18,328 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,328 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,329 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,330 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:18,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:18,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,333 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,334 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,334 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,334 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:18,334 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:18,334 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,434 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:18,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:18,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:18,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:18,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:18,437 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:18,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:18,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:18,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,437 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,440 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:18,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:18,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:18,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:18,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:18,442 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,442 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:18,442 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:18,442 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,442 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:18,442 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:18,443 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,443 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:18,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b10b0d8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:18,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:18,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,446 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:18,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:18,446 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:18,446 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,446 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:18,447 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,447 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:18,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,448 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:18,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:18,449 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:18,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:18,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:18,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:18,450 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:18,452 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,452 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,452 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,453 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,454 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,458 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:18,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:18,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:18,458 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:18,458 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:18,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/324887755ae5455284724d9a79e4bd90 2023-04-16 20:55:18,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/25b21c6e1bcd4d039e76fca6813a51ac as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/25b21c6e1bcd4d039e76fca6813a51ac 2023-04-16 20:55:18,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/25b21c6e1bcd4d039e76fca6813a51ac, entries=200, sequenceid=322, filesize=32.3 K 2023-04-16 20:55:18,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/9c95063bb6484ce79104e2dbab885dd4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9c95063bb6484ce79104e2dbab885dd4 2023-04-16 20:55:18,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9c95063bb6484ce79104e2dbab885dd4, entries=150, sequenceid=322, filesize=11.9 K 2023-04-16 20:55:18,478 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/324887755ae5455284724d9a79e4bd90 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/324887755ae5455284724d9a79e4bd90 2023-04-16 20:55:18,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/324887755ae5455284724d9a79e4bd90, entries=150, sequenceid=322, filesize=11.9 K 2023-04-16 20:55:18,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 1e76395fa47d06dccc999b09513e8aa1 in 902ms, sequenceid=322, compaction requested=true 2023-04-16 20:55:18,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:18,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:18,482 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:55:18,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:18,482 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:55:18,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:18,485 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 156262 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:55:18,485 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/A is initiating minor compaction (all files) 2023-04-16 20:55:18,485 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/A in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,485 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/1522526975094c1ab9d59598109a5877, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/66380952164b4beda5f6b82712084bd6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/3123fc812dbf4997b4022bdfb6e5fd50, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/2d999dff8cc944d890beb92cde812321, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/8d27093da8884c8c906fd7b9bd411320, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/25b21c6e1bcd4d039e76fca6813a51ac] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=152.6 K 2023-04-16 20:55:18,485 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70666 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:55:18,485 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/B is initiating minor compaction (all files) 2023-04-16 20:55:18,485 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/B in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,485 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/8162bf7f2b9e489c80dbaf7bda4897f9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/0a9462116cd54a8cb4718e049c43c855, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1be54ca1494649a8bf28aad47e8df9c7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/6709dd3730ff4eb784b200f5649f85cc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d4723fc16025444cadb0715119be957b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9c95063bb6484ce79104e2dbab885dd4] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=69.0 K 2023-04-16 20:55:18,485 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1522526975094c1ab9d59598109a5877, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1681678511815 2023-04-16 20:55:18,485 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8162bf7f2b9e489c80dbaf7bda4897f9, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1681678511815 2023-04-16 20:55:18,486 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 66380952164b4beda5f6b82712084bd6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681678511979 2023-04-16 20:55:18,486 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0a9462116cd54a8cb4718e049c43c855, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681678511979 2023-04-16 20:55:18,486 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3123fc812dbf4997b4022bdfb6e5fd50, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681678513161 2023-04-16 20:55:18,486 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1be54ca1494649a8bf28aad47e8df9c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681678513161 2023-04-16 20:55:18,487 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6709dd3730ff4eb784b200f5649f85cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681678515391 2023-04-16 20:55:18,487 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2d999dff8cc944d890beb92cde812321, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681678515391 2023-04-16 20:55:18,487 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8d27093da8884c8c906fd7b9bd411320, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681678516032 2023-04-16 20:55:18,488 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d4723fc16025444cadb0715119be957b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681678516032 2023-04-16 20:55:18,488 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 25b21c6e1bcd4d039e76fca6813a51ac, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681678517319 2023-04-16 20:55:18,488 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9c95063bb6484ce79104e2dbab885dd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681678517327 2023-04-16 20:55:18,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:18,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:55:18,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:18,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:18,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:18,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:18,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:18,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:18,505 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#B#compaction#387 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:18,518 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#A#compaction#388 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:18,547 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/94963662b30e490e92555fe1a69762be as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/94963662b30e490e92555fe1a69762be 2023-04-16 20:55:18,551 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/B of 1e76395fa47d06dccc999b09513e8aa1 into 94963662b30e490e92555fe1a69762be(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:18,551 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:18,551 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/B, priority=10, startTime=1681678518482; duration=0sec 2023-04-16 20:55:18,551 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:18,551 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:55:18,552 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416eb49c5d4009d4704bb50790b10c27cdc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416eb49c5d4009d4704bb50790b10c27cdc 2023-04-16 20:55:18,554 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70666 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:55:18,554 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/C is initiating minor compaction (all files) 2023-04-16 20:55:18,554 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/C in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,554 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/e8a3ea3d0a2e48c0b991f56cf398fe33, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/93a7813e34e24fb2af1264555be13b5f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/edfb46f78e6b49a9a27d09c5484b0b69, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/fdda4a882df146f1acad01498a828d4b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/9bf2cf418fcf443e9326b18d59bee61e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/324887755ae5455284724d9a79e4bd90] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=69.0 K 2023-04-16 20:55:18,554 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e8a3ea3d0a2e48c0b991f56cf398fe33, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1681678511815 2023-04-16 20:55:18,555 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 93a7813e34e24fb2af1264555be13b5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681678511979 2023-04-16 20:55:18,555 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting edfb46f78e6b49a9a27d09c5484b0b69, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681678513161 2023-04-16 20:55:18,555 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting fdda4a882df146f1acad01498a828d4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681678515391 2023-04-16 20:55:18,556 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9bf2cf418fcf443e9326b18d59bee61e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681678516032 2023-04-16 20:55:18,556 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 324887755ae5455284724d9a79e4bd90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681678517327 2023-04-16 20:55:18,558 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:18,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:18,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:18,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:18,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:18,566 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:18,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,566 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#C#compaction#389 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:18,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:18,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:18,567 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,567 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,568 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,569 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:18,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:18,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:18,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:18,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,576 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:18,579 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,579 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:18,579 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:18,579 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:18,581 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,581 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,581 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,581 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db48f7b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:18,582 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:18,583 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,584 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:18,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,585 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:18,585 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:18,585 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:18,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:18,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,586 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:18,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:18,588 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:18,588 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:18,588 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:18,588 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:18,588 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:18,589 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,589 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,589 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,590 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,591 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,596 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,596 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,596 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:18,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:18,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:18,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,597 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:18,596 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,596 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,596 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,597 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:18,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,597 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678518600, max timestamp=1681678518623Num uniques 859; 2023-04-16 20:55:18,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:18,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678518600, max timestamp=1681678518623Num uniques 859; 2023-04-16 20:55:18,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678518600, max timestamp=1681678518623Num uniques 859; 2023-04-16 20:55:18,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:18,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:18,697 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:18,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:18,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:18,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:18,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:18,700 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:18,700 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,700 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:18,700 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:18,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,701 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,703 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,703 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:18,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:18,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:18,704 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,704 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:18,704 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:18,704 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:18,706 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,706 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:18,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5782d569[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:18,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:18,707 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,708 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,708 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:18,708 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:18,709 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:18,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:18,711 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,711 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,711 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,711 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:18,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:18,713 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:18,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:18,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:18,713 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:18,713 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:18,714 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,714 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,715 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,716 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,717 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,718 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:18,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:18,718 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,719 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,719 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,719 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:18,719 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:18,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:18,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678578755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678578800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,819 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:18,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:18,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:18,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:18,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:18,822 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,823 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,823 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:18,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:18,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:18,826 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:18,826 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:18,826 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,826 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:18,826 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:18,826 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:18,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:18,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7be8604a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:18,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:18,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,835 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,835 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:18,835 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:18,835 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:18,836 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,836 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:18,837 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,837 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:18,837 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,837 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:18,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:18,844 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:18,844 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:18,844 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:18,844 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:18,844 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:18,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,847 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,849 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,852 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,853 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,853 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:18,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:18,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:18,854 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:18,854 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:18,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:18,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678578863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,954 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:18,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:18,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:18,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:18,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:18,957 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:18,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:18,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:18,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,958 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,959 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:18,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:18,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:18,961 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,961 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:18,961 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:18,961 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:18,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:18,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:18,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:18,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:18,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:18,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4540899f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:18,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:18,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,967 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:18,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:18,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,973 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=332, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/9cd799f289bf4fafa17bfd973a0c4e94 2023-04-16 20:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:18,975 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:18,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:18,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:18,976 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:18,976 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:18,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:18,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,980 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,981 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:18,984 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/b3b0607f7fad4635af3958ec5358e8c9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b3b0607f7fad4635af3958ec5358e8c9 2023-04-16 20:55:18,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:18,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:18,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:18,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:18,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:18,985 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:18,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:18,986 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:18,986 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:18,986 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:18,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:18,986 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:18,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:18,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:18,990 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/C of 1e76395fa47d06dccc999b09513e8aa1 into b3b0607f7fad4635af3958ec5358e8c9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:18,990 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:18,990 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/C, priority=10, startTime=1681678518482; duration=0sec 2023-04-16 20:55:18,990 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:19,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:19,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678579069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,086 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:19,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:19,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:19,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:19,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:19,089 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:19,089 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:19,089 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:19,089 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:19,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,092 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:19,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:19,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:19,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:19,093 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:19,094 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:19,094 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:19,094 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:19,094 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:19,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,095 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c76191b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:19,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:19,097 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,098 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:19,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:19,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:19,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:19,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,099 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:19,102 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:19,102 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:19,102 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:19,102 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:19,102 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:19,104 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,104 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:19,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,104 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:19,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:19,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,108 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:19,108 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:19,108 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:19,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:19,108 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,108 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:19,108 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:19,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:19,108 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:19,108 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:19,209 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:19,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:19,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:19,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:19,212 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:19,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:19,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:19,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:19,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:19,213 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,214 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,214 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:19,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:19,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:19,215 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:19,217 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,217 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:19,217 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,217 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d1eba61[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:19,218 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:19,218 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,220 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:19,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:19,221 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:19,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:19,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:19,223 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:19,223 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:19,224 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:19,224 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:19,224 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:19,224 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:19,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,225 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:19,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:19,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,227 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,229 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:19,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:19,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:19,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:19,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:19,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:19,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:19,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,229 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:19,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:19,229 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:19,229 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:19,329 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:19,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:19,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:19,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:19,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:19,333 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,336 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:19,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:19,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,336 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:19,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:19,337 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:19,337 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:19,337 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:19,337 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:19,339 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cc7fb8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:19,342 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:19,342 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,343 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:19,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:19,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:19,344 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,344 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:19,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:19,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:19,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:19,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:19,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:19,348 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:19,348 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:19,348 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:19,348 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:19,348 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:19,351 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,351 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,351 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:19,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:19,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:19,352 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,354 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:19,356 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:19,357 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:19,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:19,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:19,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:19,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:19,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:19,357 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:19,357 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:19,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:19,363 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/92c2c80021034af297b70b4921c3b299 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/92c2c80021034af297b70b4921c3b299 2023-04-16 20:55:19,367 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/A of 1e76395fa47d06dccc999b09513e8aa1 into 92c2c80021034af297b70b4921c3b299(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:19,367 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:19,367 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/A, priority=10, startTime=1681678518482; duration=0sec 2023-04-16 20:55:19,367 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:19,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:19,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678579378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/3b403b7af2c244058f4d14d39276d672 2023-04-16 20:55:19,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/05527b62f0a9455b936f8db390a401a1 2023-04-16 20:55:19,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/9cd799f289bf4fafa17bfd973a0c4e94 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9cd799f289bf4fafa17bfd973a0c4e94 2023-04-16 20:55:19,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9cd799f289bf4fafa17bfd973a0c4e94, entries=200, sequenceid=332, filesize=32.3 K 2023-04-16 20:55:19,408 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/3b403b7af2c244058f4d14d39276d672 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3b403b7af2c244058f4d14d39276d672 2023-04-16 20:55:19,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3b403b7af2c244058f4d14d39276d672, entries=150, sequenceid=332, filesize=11.9 K 2023-04-16 20:55:19,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/05527b62f0a9455b936f8db390a401a1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/05527b62f0a9455b936f8db390a401a1 2023-04-16 20:55:19,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/05527b62f0a9455b936f8db390a401a1, entries=150, sequenceid=332, filesize=11.9 K 2023-04-16 20:55:19,416 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1e76395fa47d06dccc999b09513e8aa1 in 921ms, sequenceid=332, compaction requested=false 2023-04-16 20:55:19,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:19,457 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:19,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:19,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:19,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:19,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:19,460 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:19,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:19,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:19,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:19,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,460 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,461 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:19,461 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:19,462 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,462 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:19,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:19,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:19,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:19,463 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:19,463 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:19,463 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:19,463 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:19,463 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:19,463 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:19,463 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:19,463 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 20:55:19,463 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:19,463 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:19,463 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:19,463 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:19,463 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:19,464 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:19,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:19,813 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:19,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:19,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678579863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,877 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416cc9fc0ae7dba4ee8be71e746b3032d53 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416cc9fc0ae7dba4ee8be71e746b3032d53 2023-04-16 20:55:19,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:19,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678579886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:19,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:19,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678579967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:20,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:20,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678580172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:20,282 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/9a3ce03edec44f4f8e8914d21f76e0ae 2023-04-16 20:55:20,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:20,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678580476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:20,693 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/1b1132709cd4486b84c8d76149c8e0ef 2023-04-16 20:55:20,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:20,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678580896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:20,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678580985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,104 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/c6f6e3f4015a4a568b5cb35e30f34270 2023-04-16 20:55:21,108 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/9a3ce03edec44f4f8e8914d21f76e0ae as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9a3ce03edec44f4f8e8914d21f76e0ae 2023-04-16 20:55:21,111 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9a3ce03edec44f4f8e8914d21f76e0ae, entries=150, sequenceid=372, filesize=25.4 K 2023-04-16 20:55:21,112 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/1b1132709cd4486b84c8d76149c8e0ef as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1b1132709cd4486b84c8d76149c8e0ef 2023-04-16 20:55:21,115 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1b1132709cd4486b84c8d76149c8e0ef, entries=150, sequenceid=372, filesize=11.9 K 2023-04-16 20:55:21,116 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/c6f6e3f4015a4a568b5cb35e30f34270 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c6f6e3f4015a4a568b5cb35e30f34270 2023-04-16 20:55:21,119 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c6f6e3f4015a4a568b5cb35e30f34270, entries=150, sequenceid=372, filesize=11.9 K 2023-04-16 20:55:21,119 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 1e76395fa47d06dccc999b09513e8aa1 in 1656ms, sequenceid=372, compaction requested=true 2023-04-16 20:55:21,119 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:21,119 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:21,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:21,126 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,126 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:21,126 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:21,126 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:21,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:21,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4811aef[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:21,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:21,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:21,129 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:21,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:21,129 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,129 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:21,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:21,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:21,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:21,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:21,133 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:21,133 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:21,133 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:21,133 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:21,133 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:21,136 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:21,136 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:21,136 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:21,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:21,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:21,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:21,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:21,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:21,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:21,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:21,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:21,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:21,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:21,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:21,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:21,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:21,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:21,141 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:21,141 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:21,241 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:21,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:21,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:21,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:21,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:21,244 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:21,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:21,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:21,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:21,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:21,245 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:21,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:21,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:21,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:21,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:21,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:21,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:21,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:21,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:21,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:21,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:21,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:21,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:21,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:21,247 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:21,247 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:21,248 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:55:21,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:21,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:21,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:21,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:21,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:21,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:21,260 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416f1b4a26a92e54c159c81338b853c4607 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f1b4a26a92e54c159c81338b853c4607 2023-04-16 20:55:21,665 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=382, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/9ab61b867f46471b91de04b54832f619 2023-04-16 20:55:22,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:22,013 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:22,076 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/671bbd6f4179408d86d9bc92451ff430 2023-04-16 20:55:22,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678522142, max timestamp=1681678522167Num uniques 859; 2023-04-16 20:55:22,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678522142, max timestamp=1681678522167Num uniques 859; 2023-04-16 20:55:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678522142, max timestamp=1681678522167Num uniques 859; 2023-04-16 20:55:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:22,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:22,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678582273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:22,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678582381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,506 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/b41a9f481a824a6fb23be38c8b8ad8af 2023-04-16 20:55:22,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/9ab61b867f46471b91de04b54832f619 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9ab61b867f46471b91de04b54832f619 2023-04-16 20:55:22,513 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9ab61b867f46471b91de04b54832f619, entries=100, sequenceid=382, filesize=18.6 K 2023-04-16 20:55:22,513 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/671bbd6f4179408d86d9bc92451ff430 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/671bbd6f4179408d86d9bc92451ff430 2023-04-16 20:55:22,516 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/671bbd6f4179408d86d9bc92451ff430, entries=100, sequenceid=382, filesize=9.5 K 2023-04-16 20:55:22,517 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/b41a9f481a824a6fb23be38c8b8ad8af as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b41a9f481a824a6fb23be38c8b8ad8af 2023-04-16 20:55:22,520 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b41a9f481a824a6fb23be38c8b8ad8af, entries=100, sequenceid=382, filesize=9.5 K 2023-04-16 20:55:22,521 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1e76395fa47d06dccc999b09513e8aa1 in 1274ms, sequenceid=382, compaction requested=true 2023-04-16 20:55:22,521 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:22,521 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:22,522 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,522 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:22,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83375a5[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:22,524 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:22,524 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,526 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:22,526 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,526 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:22,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:22,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:22,529 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:22,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:22,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:22,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:22,530 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:22,531 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,531 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,531 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,532 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,533 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:22,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:22,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,535 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:22,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,535 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:22,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:22,535 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:22,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:22,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:22,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:22,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:22,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:22,594 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-16 20:55:22,594 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:22,594 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:55:22,594 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:22,594 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:55:22,594 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:22,594 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:55:22,635 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:22,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:22,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:22,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678582629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:22,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:22,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:22,638 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:22,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:22,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:22,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:22,639 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,639 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,641 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,641 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:22,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:22,641 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:22,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:22,643 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:22,643 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:22,643 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:22,643 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:22,643 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:22,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,644 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,645 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,645 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f61b70[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:22,646 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:22,646 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,647 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,647 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:22,647 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:22,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:22,647 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:22,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,649 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,649 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:22,649 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:22,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:22,651 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:22,651 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:22,651 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:22,651 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:22,651 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:22,654 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,654 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,654 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,654 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:22,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,655 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:22,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,657 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:22,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:22,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:22,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:22,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:22,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:22,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,658 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:22,658 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:22,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:22,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678582737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,758 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:22,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:22,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:22,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:22,761 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:22,761 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:22,761 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:22,761 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:22,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,763 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:22,763 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:22,763 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,763 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:22,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:22,765 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:22,768 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:22,768 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:22,768 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:22,768 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:22,770 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26618569[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:22,771 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:22,771 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,775 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:22,776 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,776 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:22,777 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:22,779 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:22,779 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:22,779 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:22,780 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:22,780 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:22,781 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,781 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,781 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:22,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:22,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:22,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:22,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:22,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:22,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:22,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:22,786 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:22,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:22,786 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:22,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:22,886 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:22,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:22,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:22,895 DEBUG [Thread-2482] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:54578 2023-04-16 20:55:22,895 DEBUG [Thread-2482] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:22,898 DEBUG [Thread-2488] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ce79633 to 127.0.0.1:54578 2023-04-16 20:55:22,898 DEBUG [Thread-2488] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:22,898 DEBUG [Thread-2486] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eb8f142 to 127.0.0.1:54578 2023-04-16 20:55:22,898 DEBUG [Thread-2486] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:22,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:22,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:22,903 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:22,904 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:22,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:22,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:22,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,905 DEBUG [Thread-2484] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:54578 2023-04-16 20:55:22,905 DEBUG [Thread-2484] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:22,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,907 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:22,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:22,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:22,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:22,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:22,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:22,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:22,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:22,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:22,910 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:22,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:22,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. started... 2023-04-16 20:55:22,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. as already flushing 2023-04-16 20:55:22,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:22,912 DEBUG [Thread-2480] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:54578 2023-04-16 20:55:22,912 DEBUG [Thread-2480] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:22,913 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,913 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,914 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,914 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:22,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2abbb528[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:22,916 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:22,916 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,919 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,919 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:22,919 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:22,919 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:22,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:22,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:22,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46756 deadline: 1681678582920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,921 DEBUG [Thread-2475] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:55:22,921 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:22,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:22,924 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:22,924 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:22,924 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:22,924 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:22,924 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:22,925 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,925 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:22,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,925 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:22,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:22,926 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,929 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:22,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:22,932 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:22,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:22,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:22,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:22,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:22,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:22,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:22,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:22,932 DEBUG [Thread-2479] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:22,932 DEBUG [Thread-2479] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:22,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:22,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678582948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:23,005 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041616423523f55d47d9954b8493108f39d8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041616423523f55d47d9954b8493108f39d8 2023-04-16 20:55:23,032 DEBUG [Thread-2479] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:23,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:23,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:23,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678583249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:23,410 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=420, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/99ca73b485054b4eb9e2e8de65790622 2023-04-16 20:55:23,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46736 deadline: 1681678583750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:23,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/91eff14ae1ea4628817999c96970497b 2023-04-16 20:55:24,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/c458b0aae95a48df8152432f925ea860 2023-04-16 20:55:24,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/99ca73b485054b4eb9e2e8de65790622 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/99ca73b485054b4eb9e2e8de65790622 2023-04-16 20:55:24,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/99ca73b485054b4eb9e2e8de65790622, entries=200, sequenceid=420, filesize=32.3 K 2023-04-16 20:55:24,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/91eff14ae1ea4628817999c96970497b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/91eff14ae1ea4628817999c96970497b 2023-04-16 20:55:24,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/91eff14ae1ea4628817999c96970497b, entries=150, sequenceid=420, filesize=11.9 K 2023-04-16 20:55:24,236 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/c458b0aae95a48df8152432f925ea860 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c458b0aae95a48df8152432f925ea860 2023-04-16 20:55:24,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c458b0aae95a48df8152432f925ea860, entries=150, sequenceid=420, filesize=11.9 K 2023-04-16 20:55:24,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 1e76395fa47d06dccc999b09513e8aa1 in 1647ms, sequenceid=420, compaction requested=true 2023-04-16 20:55:24,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:24,240 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:24,240 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:24,240 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:55:24,240 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:55:24,240 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:24,241 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137910 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:55:24,241 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:55:24,241 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/A is initiating minor compaction (all files) 2023-04-16 20:55:24,241 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/B is initiating minor compaction (all files) 2023-04-16 20:55:24,241 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/A in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:24,241 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/B in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:24,242 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/92c2c80021034af297b70b4921c3b299, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9cd799f289bf4fafa17bfd973a0c4e94, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9a3ce03edec44f4f8e8914d21f76e0ae, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9ab61b867f46471b91de04b54832f619, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/99ca73b485054b4eb9e2e8de65790622] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=134.7 K 2023-04-16 20:55:24,242 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/94963662b30e490e92555fe1a69762be, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3b403b7af2c244058f4d14d39276d672, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1b1132709cd4486b84c8d76149c8e0ef, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/671bbd6f4179408d86d9bc92451ff430, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/91eff14ae1ea4628817999c96970497b] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=57.7 K 2023-04-16 20:55:24,242 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 92c2c80021034af297b70b4921c3b299, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681678517327 2023-04-16 20:55:24,242 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 94963662b30e490e92555fe1a69762be, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681678517327 2023-04-16 20:55:24,242 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9cd799f289bf4fafa17bfd973a0c4e94, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681678517604 2023-04-16 20:55:24,242 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3b403b7af2c244058f4d14d39276d672, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681678517634 2023-04-16 20:55:24,242 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9a3ce03edec44f4f8e8914d21f76e0ae, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681678518723 2023-04-16 20:55:24,242 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1b1132709cd4486b84c8d76149c8e0ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681678518723 2023-04-16 20:55:24,243 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9ab61b867f46471b91de04b54832f619, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681678519855 2023-04-16 20:55:24,243 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 671bbd6f4179408d86d9bc92451ff430, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681678519855 2023-04-16 20:55:24,243 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 99ca73b485054b4eb9e2e8de65790622, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681678522239 2023-04-16 20:55:24,243 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 91eff14ae1ea4628817999c96970497b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681678522247 2023-04-16 20:55:24,249 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#B#compaction#402 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:24,252 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#A#compaction#401 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:24,658 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/7744e0e23406400cb5d958882fb058a5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/7744e0e23406400cb5d958882fb058a5 2023-04-16 20:55:24,662 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/B of 1e76395fa47d06dccc999b09513e8aa1 into 7744e0e23406400cb5d958882fb058a5(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:24,662 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:24,662 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/B, priority=11, startTime=1681678524240; duration=0sec 2023-04-16 20:55:24,662 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:24,662 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:55:24,663 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:55:24,663 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 1e76395fa47d06dccc999b09513e8aa1/C is initiating minor compaction (all files) 2023-04-16 20:55:24,663 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1e76395fa47d06dccc999b09513e8aa1/C in TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:24,664 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b3b0607f7fad4635af3958ec5358e8c9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/05527b62f0a9455b936f8db390a401a1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c6f6e3f4015a4a568b5cb35e30f34270, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b41a9f481a824a6fb23be38c8b8ad8af, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c458b0aae95a48df8152432f925ea860] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp, totalSize=57.7 K 2023-04-16 20:55:24,664 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b3b0607f7fad4635af3958ec5358e8c9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681678517327 2023-04-16 20:55:24,664 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 05527b62f0a9455b936f8db390a401a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681678517634 2023-04-16 20:55:24,664 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c6f6e3f4015a4a568b5cb35e30f34270, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681678518723 2023-04-16 20:55:24,665 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b41a9f481a824a6fb23be38c8b8ad8af, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681678519855 2023-04-16 20:55:24,665 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c458b0aae95a48df8152432f925ea860, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681678522247 2023-04-16 20:55:24,672 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 1e76395fa47d06dccc999b09513e8aa1#C#compaction#403 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:24,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:24,753 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:55:24,753 DEBUG [Thread-2469] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:54578 2023-04-16 20:55:24,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:24,753 DEBUG [Thread-2469] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:24,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:24,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:24,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:24,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:24,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:25,079 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/94c0c5e64e1240d68f7b8eb49a362d19 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/94c0c5e64e1240d68f7b8eb49a362d19 2023-04-16 20:55:25,082 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/C of 1e76395fa47d06dccc999b09513e8aa1 into 94c0c5e64e1240d68f7b8eb49a362d19(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:25,082 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:25,082 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/C, priority=11, startTime=1681678524240; duration=0sec 2023-04-16 20:55:25,082 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:25,163 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304165bf6711268db431db4a59a2a079a6962 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165bf6711268db431db4a59a2a079a6962 2023-04-16 20:55:25,469 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/44097f2d293649b5a2981fbfe8be4585 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/44097f2d293649b5a2981fbfe8be4585 2023-04-16 20:55:25,472 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1e76395fa47d06dccc999b09513e8aa1/A of 1e76395fa47d06dccc999b09513e8aa1 into 44097f2d293649b5a2981fbfe8be4585(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-16 20:55:25,473 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:25,473 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1., storeName=1e76395fa47d06dccc999b09513e8aa1/A, priority=11, startTime=1681678524240; duration=1sec 2023-04-16 20:55:25,473 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:25,567 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=431, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/1dfd6b9c234941c3a3df8bff67def4c6 2023-04-16 20:55:25,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/893dcf902d39438b8af98e1ee0a96499 2023-04-16 20:55:26,331 DEBUG [Thread-2477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:54578 2023-04-16 20:55:26,331 DEBUG [Thread-2477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:26,364 DEBUG [Thread-2473] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:54578 2023-04-16 20:55:26,364 DEBUG [Thread-2473] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:26,383 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/351e57b5e7bd40029bfc2d83df101e13 2023-04-16 20:55:26,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/1dfd6b9c234941c3a3df8bff67def4c6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/1dfd6b9c234941c3a3df8bff67def4c6 2023-04-16 20:55:26,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/1dfd6b9c234941c3a3df8bff67def4c6, entries=150, sequenceid=431, filesize=25.4 K 2023-04-16 20:55:26,390 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/893dcf902d39438b8af98e1ee0a96499 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/893dcf902d39438b8af98e1ee0a96499 2023-04-16 20:55:26,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/893dcf902d39438b8af98e1ee0a96499, entries=150, sequenceid=431, filesize=11.9 K 2023-04-16 20:55:26,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/351e57b5e7bd40029bfc2d83df101e13 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/351e57b5e7bd40029bfc2d83df101e13 2023-04-16 20:55:26,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/351e57b5e7bd40029bfc2d83df101e13, entries=150, sequenceid=431, filesize=11.9 K 2023-04-16 20:55:26,397 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=13.42 KB/13740 for 1e76395fa47d06dccc999b09513e8aa1 in 1644ms, sequenceid=431, compaction requested=false 2023-04-16 20:55:26,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:26,413 DEBUG [Thread-2471] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:54578 2023-04-16 20:55:26,413 DEBUG [Thread-2471] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:26,960 DEBUG [Thread-2475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:54578 2023-04-16 20:55:26,960 DEBUG [Thread-2475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1837 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5509 rows 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1840 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5520 rows 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1847 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5541 rows 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1843 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5528 rows 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1840 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5520 rows 2023-04-16 20:55:26,961 INFO [Listener at localhost/45179] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 20:55:26,961 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:54578 2023-04-16 20:55:26,961 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:26,965 INFO [Listener at localhost/45179] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 20:55:26,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 20:55:26,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:26,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 20:55:26,969 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678526969"}]},"ts":"1681678526969"} 2023-04-16 20:55:26,970 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 20:55:26,971 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 20:55:26,972 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, UNASSIGN}] 2023-04-16 20:55:26,973 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, UNASSIGN 2023-04-16 20:55:26,973 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=1e76395fa47d06dccc999b09513e8aa1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:26,973 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678526973"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678526973"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678526973"}]},"ts":"1681678526973"} 2023-04-16 20:55:26,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:55:27,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 20:55:27,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1e76395fa47d06dccc999b09513e8aa1, disabling compactions & flushes 2023-04-16 20:55:27,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. after waiting 0 ms 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:27,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1e76395fa47d06dccc999b09513e8aa1 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=A 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=B 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1e76395fa47d06dccc999b09513e8aa1, store=C 2023-04-16 20:55:27,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:27,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 20:55:27,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041600ac846e0a8943b18e95f99b3246220e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041600ac846e0a8943b18e95f99b3246220e 2023-04-16 20:55:27,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 20:55:27,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=440, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/4ec68ab67ccb4d0180676e2c837aad5b 2023-04-16 20:55:28,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 20:55:28,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/859ff44ebbac423ea71d776a886ac042 2023-04-16 20:55:28,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/7b633ebb71e942d0aa594d1b07c0bf21 2023-04-16 20:55:28,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/A/4ec68ab67ccb4d0180676e2c837aad5b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4ec68ab67ccb4d0180676e2c837aad5b 2023-04-16 20:55:28,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4ec68ab67ccb4d0180676e2c837aad5b, entries=100, sequenceid=440, filesize=18.6 K 2023-04-16 20:55:28,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/B/859ff44ebbac423ea71d776a886ac042 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/859ff44ebbac423ea71d776a886ac042 2023-04-16 20:55:28,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/859ff44ebbac423ea71d776a886ac042, entries=100, sequenceid=440, filesize=9.5 K 2023-04-16 20:55:28,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/.tmp/C/7b633ebb71e942d0aa594d1b07c0bf21 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7b633ebb71e942d0aa594d1b07c0bf21 2023-04-16 20:55:28,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7b633ebb71e942d0aa594d1b07c0bf21, entries=100, sequenceid=440, filesize=9.5 K 2023-04-16 20:55:28,774 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1e76395fa47d06dccc999b09513e8aa1 in 1648ms, sequenceid=440, compaction requested=true 2023-04-16 20:55:28,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ccf630370f11466bb69be68c464c891e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ec919528ef42494cbb52645f8c0a7367, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ef0d4235622d4ac38aae362009f5e701, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9166094267124ca6acfe07c2b9c81c4a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/330a8a5f1286432a8cf7cab731681cf3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/949f83899fea4f25b934dea9e91001f5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/bff209a3621e4f1d8a8bd35616e14435, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/78e03149066347ff9ef34f9c09f4a3ff, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4814e3e481f2405f8772d7796196745c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/c687c51caaa84e7e9aeaa85cdb18d060, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/435b2be23f6344d88e9403488e49ab96, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/a8fa71eab8d34baa890da2f0443bdef8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/87b3edcdb5cf4942a65a10597c0f0437, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/1522526975094c1ab9d59598109a5877, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/66380952164b4beda5f6b82712084bd6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/3123fc812dbf4997b4022bdfb6e5fd50, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/2d999dff8cc944d890beb92cde812321, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/8d27093da8884c8c906fd7b9bd411320, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/25b21c6e1bcd4d039e76fca6813a51ac, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/92c2c80021034af297b70b4921c3b299, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9cd799f289bf4fafa17bfd973a0c4e94, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9a3ce03edec44f4f8e8914d21f76e0ae, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9ab61b867f46471b91de04b54832f619, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/99ca73b485054b4eb9e2e8de65790622] to archive 2023-04-16 20:55:28,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:28,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ccf630370f11466bb69be68c464c891e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ccf630370f11466bb69be68c464c891e 2023-04-16 20:55:28,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ec919528ef42494cbb52645f8c0a7367 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ec919528ef42494cbb52645f8c0a7367 2023-04-16 20:55:28,798 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ef0d4235622d4ac38aae362009f5e701 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/ef0d4235622d4ac38aae362009f5e701 2023-04-16 20:55:28,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9166094267124ca6acfe07c2b9c81c4a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9166094267124ca6acfe07c2b9c81c4a 2023-04-16 20:55:28,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/330a8a5f1286432a8cf7cab731681cf3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/330a8a5f1286432a8cf7cab731681cf3 2023-04-16 20:55:28,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/949f83899fea4f25b934dea9e91001f5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/949f83899fea4f25b934dea9e91001f5 2023-04-16 20:55:28,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/bff209a3621e4f1d8a8bd35616e14435 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/bff209a3621e4f1d8a8bd35616e14435 2023-04-16 20:55:28,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/78e03149066347ff9ef34f9c09f4a3ff to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/78e03149066347ff9ef34f9c09f4a3ff 2023-04-16 20:55:28,803 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4814e3e481f2405f8772d7796196745c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4814e3e481f2405f8772d7796196745c 2023-04-16 20:55:28,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/c687c51caaa84e7e9aeaa85cdb18d060 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/c687c51caaa84e7e9aeaa85cdb18d060 2023-04-16 20:55:28,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/435b2be23f6344d88e9403488e49ab96 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/435b2be23f6344d88e9403488e49ab96 2023-04-16 20:55:28,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/a8fa71eab8d34baa890da2f0443bdef8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/a8fa71eab8d34baa890da2f0443bdef8 2023-04-16 20:55:28,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/87b3edcdb5cf4942a65a10597c0f0437 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/87b3edcdb5cf4942a65a10597c0f0437 2023-04-16 20:55:28,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/1522526975094c1ab9d59598109a5877 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/1522526975094c1ab9d59598109a5877 2023-04-16 20:55:28,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/66380952164b4beda5f6b82712084bd6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/66380952164b4beda5f6b82712084bd6 2023-04-16 20:55:28,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/3123fc812dbf4997b4022bdfb6e5fd50 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/3123fc812dbf4997b4022bdfb6e5fd50 2023-04-16 20:55:28,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/2d999dff8cc944d890beb92cde812321 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/2d999dff8cc944d890beb92cde812321 2023-04-16 20:55:28,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/8d27093da8884c8c906fd7b9bd411320 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/8d27093da8884c8c906fd7b9bd411320 2023-04-16 20:55:28,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/25b21c6e1bcd4d039e76fca6813a51ac to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/25b21c6e1bcd4d039e76fca6813a51ac 2023-04-16 20:55:28,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/92c2c80021034af297b70b4921c3b299 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/92c2c80021034af297b70b4921c3b299 2023-04-16 20:55:28,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9cd799f289bf4fafa17bfd973a0c4e94 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9cd799f289bf4fafa17bfd973a0c4e94 2023-04-16 20:55:28,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9a3ce03edec44f4f8e8914d21f76e0ae to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9a3ce03edec44f4f8e8914d21f76e0ae 2023-04-16 20:55:28,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9ab61b867f46471b91de04b54832f619 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/9ab61b867f46471b91de04b54832f619 2023-04-16 20:55:28,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/99ca73b485054b4eb9e2e8de65790622 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/99ca73b485054b4eb9e2e8de65790622 2023-04-16 20:55:28,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/192ce44a06534468832fe1cbad1cf25e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/b1bce80ecfa84eaaacc746d79ca83867, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/ee2c9f0cf84a41488d7ba068bd9fe39b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/722361040ffa4630816c3e5664ccf92f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/956e8e9b2ebc4cd3acd741cd13d200b8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/e7814a62d9ee43a38251dff6fc466d56, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/fa2eff94ad30433b8815b5d30d990fff, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d918056637fd414dac5930183b6b1d94, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/876801c2fd85469ca7e82f53b54f71a4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/033682834de44970b6ef6b137d0f0e03, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/7120f8842db04ee8a1c35b1c782cf81b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3d10bab05d0b45bfb155ff04d178c8f9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/8162bf7f2b9e489c80dbaf7bda4897f9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9411d04887a34819a85aa257082db4e5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/0a9462116cd54a8cb4718e049c43c855, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1be54ca1494649a8bf28aad47e8df9c7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/6709dd3730ff4eb784b200f5649f85cc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d4723fc16025444cadb0715119be957b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/94963662b30e490e92555fe1a69762be, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9c95063bb6484ce79104e2dbab885dd4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3b403b7af2c244058f4d14d39276d672, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1b1132709cd4486b84c8d76149c8e0ef, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/671bbd6f4179408d86d9bc92451ff430, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/91eff14ae1ea4628817999c96970497b] to archive 2023-04-16 20:55:28,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:28,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/192ce44a06534468832fe1cbad1cf25e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/192ce44a06534468832fe1cbad1cf25e 2023-04-16 20:55:28,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/b1bce80ecfa84eaaacc746d79ca83867 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/b1bce80ecfa84eaaacc746d79ca83867 2023-04-16 20:55:28,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/ee2c9f0cf84a41488d7ba068bd9fe39b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/ee2c9f0cf84a41488d7ba068bd9fe39b 2023-04-16 20:55:28,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/722361040ffa4630816c3e5664ccf92f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/722361040ffa4630816c3e5664ccf92f 2023-04-16 20:55:28,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/956e8e9b2ebc4cd3acd741cd13d200b8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/956e8e9b2ebc4cd3acd741cd13d200b8 2023-04-16 20:55:28,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/e7814a62d9ee43a38251dff6fc466d56 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/e7814a62d9ee43a38251dff6fc466d56 2023-04-16 20:55:28,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/fa2eff94ad30433b8815b5d30d990fff to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/fa2eff94ad30433b8815b5d30d990fff 2023-04-16 20:55:28,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d918056637fd414dac5930183b6b1d94 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d918056637fd414dac5930183b6b1d94 2023-04-16 20:55:28,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/876801c2fd85469ca7e82f53b54f71a4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/876801c2fd85469ca7e82f53b54f71a4 2023-04-16 20:55:28,828 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/033682834de44970b6ef6b137d0f0e03 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/033682834de44970b6ef6b137d0f0e03 2023-04-16 20:55:28,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/7120f8842db04ee8a1c35b1c782cf81b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/7120f8842db04ee8a1c35b1c782cf81b 2023-04-16 20:55:28,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3d10bab05d0b45bfb155ff04d178c8f9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3d10bab05d0b45bfb155ff04d178c8f9 2023-04-16 20:55:28,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/8162bf7f2b9e489c80dbaf7bda4897f9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/8162bf7f2b9e489c80dbaf7bda4897f9 2023-04-16 20:55:28,831 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9411d04887a34819a85aa257082db4e5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9411d04887a34819a85aa257082db4e5 2023-04-16 20:55:28,832 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/0a9462116cd54a8cb4718e049c43c855 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/0a9462116cd54a8cb4718e049c43c855 2023-04-16 20:55:28,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1be54ca1494649a8bf28aad47e8df9c7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1be54ca1494649a8bf28aad47e8df9c7 2023-04-16 20:55:28,834 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/6709dd3730ff4eb784b200f5649f85cc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/6709dd3730ff4eb784b200f5649f85cc 2023-04-16 20:55:28,835 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d4723fc16025444cadb0715119be957b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/d4723fc16025444cadb0715119be957b 2023-04-16 20:55:28,835 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/94963662b30e490e92555fe1a69762be to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/94963662b30e490e92555fe1a69762be 2023-04-16 20:55:28,836 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9c95063bb6484ce79104e2dbab885dd4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/9c95063bb6484ce79104e2dbab885dd4 2023-04-16 20:55:28,837 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3b403b7af2c244058f4d14d39276d672 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/3b403b7af2c244058f4d14d39276d672 2023-04-16 20:55:28,838 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1b1132709cd4486b84c8d76149c8e0ef to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/1b1132709cd4486b84c8d76149c8e0ef 2023-04-16 20:55:28,839 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/671bbd6f4179408d86d9bc92451ff430 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/671bbd6f4179408d86d9bc92451ff430 2023-04-16 20:55:28,840 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/91eff14ae1ea4628817999c96970497b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/91eff14ae1ea4628817999c96970497b 2023-04-16 20:55:28,841 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ecc9aa2b9f0542bba8d1e273678acad3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/0508310058ca4c9ca671ffaa36ab4dac, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f61c987d0c494cb88daefd84994db3bd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/805e29240d6e47e9a6ce0dc55eb56017, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ccbaeefd841b47469632f8378e434d8e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c16740a9b5104898a6595c105d17585a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/90db54c4b766441a953bfea4fa29ddf9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ab3c443ce2a54ef68b4e0e095749959e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/029469ca8f24496890184bd68de492cf, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/1447779e86c143aa9f21e41f1b432fb2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7d983354f2d54001bc59f5e18b111c3a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/2acc2b515aca48b9b20222a56e977afb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/e8a3ea3d0a2e48c0b991f56cf398fe33, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f4c4384fddb642f7a0ef08f0ea581eeb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/93a7813e34e24fb2af1264555be13b5f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/edfb46f78e6b49a9a27d09c5484b0b69, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/fdda4a882df146f1acad01498a828d4b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/9bf2cf418fcf443e9326b18d59bee61e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b3b0607f7fad4635af3958ec5358e8c9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/324887755ae5455284724d9a79e4bd90, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/05527b62f0a9455b936f8db390a401a1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c6f6e3f4015a4a568b5cb35e30f34270, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b41a9f481a824a6fb23be38c8b8ad8af, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c458b0aae95a48df8152432f925ea860] to archive 2023-04-16 20:55:28,842 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:28,843 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ecc9aa2b9f0542bba8d1e273678acad3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ecc9aa2b9f0542bba8d1e273678acad3 2023-04-16 20:55:28,844 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/0508310058ca4c9ca671ffaa36ab4dac to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/0508310058ca4c9ca671ffaa36ab4dac 2023-04-16 20:55:28,845 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f61c987d0c494cb88daefd84994db3bd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f61c987d0c494cb88daefd84994db3bd 2023-04-16 20:55:28,845 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/805e29240d6e47e9a6ce0dc55eb56017 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/805e29240d6e47e9a6ce0dc55eb56017 2023-04-16 20:55:28,846 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ccbaeefd841b47469632f8378e434d8e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ccbaeefd841b47469632f8378e434d8e 2023-04-16 20:55:28,847 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c16740a9b5104898a6595c105d17585a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c16740a9b5104898a6595c105d17585a 2023-04-16 20:55:28,847 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/90db54c4b766441a953bfea4fa29ddf9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/90db54c4b766441a953bfea4fa29ddf9 2023-04-16 20:55:28,848 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ab3c443ce2a54ef68b4e0e095749959e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/ab3c443ce2a54ef68b4e0e095749959e 2023-04-16 20:55:28,849 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/029469ca8f24496890184bd68de492cf to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/029469ca8f24496890184bd68de492cf 2023-04-16 20:55:28,850 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/1447779e86c143aa9f21e41f1b432fb2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/1447779e86c143aa9f21e41f1b432fb2 2023-04-16 20:55:28,850 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7d983354f2d54001bc59f5e18b111c3a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7d983354f2d54001bc59f5e18b111c3a 2023-04-16 20:55:28,851 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/2acc2b515aca48b9b20222a56e977afb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/2acc2b515aca48b9b20222a56e977afb 2023-04-16 20:55:28,852 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/e8a3ea3d0a2e48c0b991f56cf398fe33 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/e8a3ea3d0a2e48c0b991f56cf398fe33 2023-04-16 20:55:28,853 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f4c4384fddb642f7a0ef08f0ea581eeb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/f4c4384fddb642f7a0ef08f0ea581eeb 2023-04-16 20:55:28,853 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/93a7813e34e24fb2af1264555be13b5f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/93a7813e34e24fb2af1264555be13b5f 2023-04-16 20:55:28,854 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/edfb46f78e6b49a9a27d09c5484b0b69 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/edfb46f78e6b49a9a27d09c5484b0b69 2023-04-16 20:55:28,855 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/fdda4a882df146f1acad01498a828d4b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/fdda4a882df146f1acad01498a828d4b 2023-04-16 20:55:28,856 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/9bf2cf418fcf443e9326b18d59bee61e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/9bf2cf418fcf443e9326b18d59bee61e 2023-04-16 20:55:28,856 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b3b0607f7fad4635af3958ec5358e8c9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b3b0607f7fad4635af3958ec5358e8c9 2023-04-16 20:55:28,857 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/324887755ae5455284724d9a79e4bd90 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/324887755ae5455284724d9a79e4bd90 2023-04-16 20:55:28,858 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/05527b62f0a9455b936f8db390a401a1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/05527b62f0a9455b936f8db390a401a1 2023-04-16 20:55:28,859 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c6f6e3f4015a4a568b5cb35e30f34270 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c6f6e3f4015a4a568b5cb35e30f34270 2023-04-16 20:55:28,860 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b41a9f481a824a6fb23be38c8b8ad8af to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/b41a9f481a824a6fb23be38c8b8ad8af 2023-04-16 20:55:28,861 DEBUG [StoreCloser-TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c458b0aae95a48df8152432f925ea860 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/c458b0aae95a48df8152432f925ea860 2023-04-16 20:55:28,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/recovered.edits/443.seqid, newMaxSeqId=443, maxSeqId=4 2023-04-16 20:55:28,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1. 2023-04-16 20:55:28,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1e76395fa47d06dccc999b09513e8aa1: 2023-04-16 20:55:28,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:28,866 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=1e76395fa47d06dccc999b09513e8aa1, regionState=CLOSED 2023-04-16 20:55:28,866 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678528866"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678528866"}]},"ts":"1681678528866"} 2023-04-16 20:55:28,868 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-16 20:55:28,868 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 1e76395fa47d06dccc999b09513e8aa1, server=jenkins-hbase4.apache.org,41211,1681678416093 in 1.8930 sec 2023-04-16 20:55:28,869 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-16 20:55:28,869 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1e76395fa47d06dccc999b09513e8aa1, UNASSIGN in 1.8960 sec 2023-04-16 20:55:28,870 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678528869"}]},"ts":"1681678528869"} 2023-04-16 20:55:28,870 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 20:55:28,871 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 20:55:28,873 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9060 sec 2023-04-16 20:55:29,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 20:55:29,073 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-16 20:55:29,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 20:55:29,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:29,076 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:29,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-16 20:55:29,077 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:29,081 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:29,084 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/recovered.edits] 2023-04-16 20:55:29,086 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/1dfd6b9c234941c3a3df8bff67def4c6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/1dfd6b9c234941c3a3df8bff67def4c6 2023-04-16 20:55:29,087 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/44097f2d293649b5a2981fbfe8be4585 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/44097f2d293649b5a2981fbfe8be4585 2023-04-16 20:55:29,089 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4ec68ab67ccb4d0180676e2c837aad5b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/A/4ec68ab67ccb4d0180676e2c837aad5b 2023-04-16 20:55:29,091 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/7744e0e23406400cb5d958882fb058a5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/7744e0e23406400cb5d958882fb058a5 2023-04-16 20:55:29,092 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/859ff44ebbac423ea71d776a886ac042 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/859ff44ebbac423ea71d776a886ac042 2023-04-16 20:55:29,093 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/893dcf902d39438b8af98e1ee0a96499 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/B/893dcf902d39438b8af98e1ee0a96499 2023-04-16 20:55:29,095 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/351e57b5e7bd40029bfc2d83df101e13 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/351e57b5e7bd40029bfc2d83df101e13 2023-04-16 20:55:29,096 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7b633ebb71e942d0aa594d1b07c0bf21 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/7b633ebb71e942d0aa594d1b07c0bf21 2023-04-16 20:55:29,098 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/94c0c5e64e1240d68f7b8eb49a362d19 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/C/94c0c5e64e1240d68f7b8eb49a362d19 2023-04-16 20:55:29,100 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/recovered.edits/443.seqid to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1/recovered.edits/443.seqid 2023-04-16 20:55:29,101 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/1e76395fa47d06dccc999b09513e8aa1 2023-04-16 20:55:29,101 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:55:29,101 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 20:55:29,102 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-16 20:55:29,105 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041600ac846e0a8943b18e95f99b3246220e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041600ac846e0a8943b18e95f99b3246220e 2023-04-16 20:55:29,106 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041611221dc25f8b452d9f7366e0037b3666 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041611221dc25f8b452d9f7366e0037b3666 2023-04-16 20:55:29,106 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041616423523f55d47d9954b8493108f39d8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041616423523f55d47d9954b8493108f39d8 2023-04-16 20:55:29,107 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161b5b7f6b9cac4ffebdac9bf46eebfea6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161b5b7f6b9cac4ffebdac9bf46eebfea6 2023-04-16 20:55:29,108 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041641cf771dea1844c592d0e8a5a6ece646 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041641cf771dea1844c592d0e8a5a6ece646 2023-04-16 20:55:29,109 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041646049ab02c174aa8b83ae745e068f229 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041646049ab02c174aa8b83ae745e068f229 2023-04-16 20:55:29,110 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165ae48715c63a4362b5bf5f4c2903eb97 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165ae48715c63a4362b5bf5f4c2903eb97 2023-04-16 20:55:29,111 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165bf6711268db431db4a59a2a079a6962 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165bf6711268db431db4a59a2a079a6962 2023-04-16 20:55:29,112 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165cd131a7b6354160b03c074418873e6b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165cd131a7b6354160b03c074418873e6b 2023-04-16 20:55:29,113 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165d2ed8fd478c45178fad1a98a1caf5d4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165d2ed8fd478c45178fad1a98a1caf5d4 2023-04-16 20:55:29,113 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165df7035025a446ad88a22490aa82529a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165df7035025a446ad88a22490aa82529a 2023-04-16 20:55:29,114 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041669b07a7bd0994560bcbc7ded23c5b0ca to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041669b07a7bd0994560bcbc7ded23c5b0ca 2023-04-16 20:55:29,115 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041676ae963757684ba4b67bb6a6257a6ec0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041676ae963757684ba4b67bb6a6257a6ec0 2023-04-16 20:55:29,116 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167d19dcdcc94649bc95640cacc8923438 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167d19dcdcc94649bc95640cacc8923438 2023-04-16 20:55:29,117 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b82f680e96d04aeab74571ee3209cbbf to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b82f680e96d04aeab74571ee3209cbbf 2023-04-16 20:55:29,117 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b91ccaa65ef54c36b1a52498da71fe23 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b91ccaa65ef54c36b1a52498da71fe23 2023-04-16 20:55:29,118 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c96e1e2620224a89a39b56fe8539a721 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c96e1e2620224a89a39b56fe8539a721 2023-04-16 20:55:29,119 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ca85afb946b144cebc63d3ab2a32ce9f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ca85afb946b144cebc63d3ab2a32ce9f 2023-04-16 20:55:29,120 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416cc9fc0ae7dba4ee8be71e746b3032d53 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416cc9fc0ae7dba4ee8be71e746b3032d53 2023-04-16 20:55:29,121 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416eb49c5d4009d4704bb50790b10c27cdc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416eb49c5d4009d4704bb50790b10c27cdc 2023-04-16 20:55:29,122 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f1b4a26a92e54c159c81338b853c4607 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f1b4a26a92e54c159c81338b853c4607 2023-04-16 20:55:29,122 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 20:55:29,124 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:29,125 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 20:55:29,126 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 20:55:29,127 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:29,127 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 20:55:29,127 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681678529127"}]},"ts":"9223372036854775807"} 2023-04-16 20:55:29,128 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 20:55:29,129 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1e76395fa47d06dccc999b09513e8aa1, NAME => 'TestAcidGuarantees,,1681678500812.1e76395fa47d06dccc999b09513e8aa1.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 20:55:29,129 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 20:55:29,129 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681678529129"}]},"ts":"9223372036854775807"} 2023-04-16 20:55:29,130 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 20:55:29,131 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:29,132 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-04-16 20:55:29,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-16 20:55:29,177 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-16 20:55:29,188 INFO [Listener at localhost/45179] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 227), OpenFileDescriptor=537 (was 542), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=395 (was 490) 2023-04-16 20:55:29,199 INFO [Listener at localhost/45179] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=537, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=169, AvailableMemoryMB=395 2023-04-16 20:55:29,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 20:55:29,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:55:29,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:29,204 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 20:55:29,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-16 20:55:29,205 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 20:55:29,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 20:55:29,206 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:29,206 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99 empty. 2023-04-16 20:55:29,207 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:29,207 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:55:29,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 20:55:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 20:55:29,615 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 20:55:29,617 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 41b019418b53f9031c0d7e1ee3de5f99, NAME => 'TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp 2023-04-16 20:55:29,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 20:55:30,034 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:55:30,034 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 41b019418b53f9031c0d7e1ee3de5f99, disabling compactions & flushes 2023-04-16 20:55:30,034 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:30,034 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:30,034 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. after waiting 0 ms 2023-04-16 20:55:30,034 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:30,034 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:30,034 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:30,036 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 20:55:30,037 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678530037"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678530037"}]},"ts":"1681678530037"} 2023-04-16 20:55:30,038 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 20:55:30,038 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 20:55:30,038 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678530038"}]},"ts":"1681678530038"} 2023-04-16 20:55:30,039 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 20:55:30,048 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41b019418b53f9031c0d7e1ee3de5f99, ASSIGN}] 2023-04-16 20:55:30,050 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41b019418b53f9031c0d7e1ee3de5f99, ASSIGN 2023-04-16 20:55:30,050 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=41b019418b53f9031c0d7e1ee3de5f99, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=false 2023-04-16 20:55:30,201 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=41b019418b53f9031c0d7e1ee3de5f99, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:30,201 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678530201"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678530201"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678530201"}]},"ts":"1681678530201"} 2023-04-16 20:55:30,203 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:55:30,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 20:55:30,357 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:30,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 41b019418b53f9031c0d7e1ee3de5f99, NAME => 'TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:55:30,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:30,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:55:30,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:30,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:30,358 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:30,360 DEBUG [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A 2023-04-16 20:55:30,360 DEBUG [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A 2023-04-16 20:55:30,360 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:30,360 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41b019418b53f9031c0d7e1ee3de5f99 columnFamilyName A 2023-04-16 20:55:30,361 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] regionserver.HStore(310): Store=41b019418b53f9031c0d7e1ee3de5f99/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:30,361 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:30,362 DEBUG [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B 2023-04-16 20:55:30,363 DEBUG [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B 2023-04-16 20:55:30,363 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:30,363 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41b019418b53f9031c0d7e1ee3de5f99 columnFamilyName B 2023-04-16 20:55:30,363 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] regionserver.HStore(310): Store=41b019418b53f9031c0d7e1ee3de5f99/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:30,363 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:30,364 DEBUG [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C 2023-04-16 20:55:30,365 DEBUG [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C 2023-04-16 20:55:30,365 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:30,365 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41b019418b53f9031c0d7e1ee3de5f99 columnFamilyName C 2023-04-16 20:55:30,365 INFO [StoreOpener-41b019418b53f9031c0d7e1ee3de5f99-1] regionserver.HStore(310): Store=41b019418b53f9031c0d7e1ee3de5f99/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:30,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:30,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:30,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:30,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:55:30,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:30,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 20:55:30,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 41b019418b53f9031c0d7e1ee3de5f99; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59381652, jitterRate=-0.11514443159103394}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:55:30,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:30,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., pid=49, masterSystemTime=1681678530354 2023-04-16 20:55:30,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:30,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:30,373 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=41b019418b53f9031c0d7e1ee3de5f99, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:30,373 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678530373"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678530373"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678530373"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678530373"}]},"ts":"1681678530373"} 2023-04-16 20:55:30,375 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-16 20:55:30,375 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 in 172 msec 2023-04-16 20:55:30,376 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-16 20:55:30,376 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=41b019418b53f9031c0d7e1ee3de5f99, ASSIGN in 327 msec 2023-04-16 20:55:30,376 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 20:55:30,376 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678530376"}]},"ts":"1681678530376"} 2023-04-16 20:55:30,377 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 20:55:30,379 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 20:55:30,380 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1770 sec 2023-04-16 20:55:31,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 20:55:31,309 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-16 20:55:31,309 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec943a9 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,315 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27f2d997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,316 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,318 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,319 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 20:55:31,320 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 20:55:31,321 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,325 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,325 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,330 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,331 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,334 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,334 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,338 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,338 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,342 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,342 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,347 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,347 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,350 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,350 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,354 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,354 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,360 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,361 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x10a344e6 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:31,365 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@720d29f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:31,376 DEBUG [hconnection-0x54040139-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,376 DEBUG [hconnection-0x9c83745-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,376 DEBUG [hconnection-0x6642b357-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,377 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36056, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,380 DEBUG [hconnection-0x4a567dd9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,380 DEBUG [hconnection-0x58fdc35e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,381 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,385 DEBUG [hconnection-0x76e2712-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,385 DEBUG [hconnection-0x539cbb5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,387 DEBUG [hconnection-0x32b11040-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,387 DEBUG [hconnection-0x587e94d6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,387 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,388 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,393 DEBUG [hconnection-0x6e33227-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:31,395 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36054, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,395 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,396 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,395 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,398 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,399 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:31,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:31,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:31,409 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:55:31,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:31,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:31,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:31,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:31,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:31,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:31,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:31,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:31,426 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:31,426 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:31,427 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:31,427 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:31,427 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,427 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,428 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:31,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:31,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:31,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,429 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,429 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:31,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:31,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:31,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681678531420, max timestamp=1681678531434Num uniques 859; 2023-04-16 20:55:31,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681678531420, max timestamp=1681678531434Num uniques 859; 2023-04-16 20:55:31,437 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-16 20:55:31,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678591432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-16 20:55:31,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681678531420, max timestamp=1681678531434Num uniques 859; 2023-04-16 20:55:31,437 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-16 20:55:31,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:31,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:31,439 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:31,439 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:31,439 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:31,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678591444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678591445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678591447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678591452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:31,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:31,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:31,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:31,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:31,456 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:31,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:31,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:31,456 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,468 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:31,468 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:31,473 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,473 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,474 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,474 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:31,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39366387[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:31,475 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:31,475 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:31,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:31,477 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,477 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:31,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:31,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:31,478 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:31,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:31,481 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:31,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:31,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:31,481 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:31,481 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:31,482 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,482 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,482 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:31,483 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,484 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:31,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:31,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:31,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:31,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:31,486 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:31,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:31,486 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:31,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,486 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:31,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,487 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,487 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678591538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678591548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678591549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678591549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678591553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,587 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:31,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:31,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:31,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:31,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:31,600 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:31,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:31,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:31,600 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:31,601 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,601 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,603 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,603 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:31,603 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,604 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:31,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:31,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:31,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:31,611 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:31,612 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:31,612 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:31,616 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:31,616 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:31,616 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:31,616 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:31,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:31,618 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,618 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,618 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,619 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:31,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@92bc24a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:31,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:31,620 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,621 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,621 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:31,621 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:31,621 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,621 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:31,621 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:31,623 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:31,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:31,625 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:31,625 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:31,625 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:31,625 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:31,625 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:31,627 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,627 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,627 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,628 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,630 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:31,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:31,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:31,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:31,632 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:31,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:31,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:31,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:31,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,632 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:31,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,633 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:31,633 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,733 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:31,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:31,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:31,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:31,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:31,740 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:31,740 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:31,740 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:31,740 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:31,740 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,740 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,742 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:31,742 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:31,742 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,742 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:31,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678591743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:31,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:31,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:31,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:31,748 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:31,748 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:31,748 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:31,748 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:31,750 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@234b54cb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:31,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:31,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678591749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678591751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678591751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,754 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,754 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:31,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:31,754 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:31,755 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:31,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:31,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:31,756 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,756 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:31,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:31,758 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:31,758 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:31,758 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:31,758 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:31,758 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:31,759 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,760 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,760 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:31,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,761 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:31,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678591755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:31,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:31,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,764 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:31,764 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:31,764 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:31,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:31,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:31,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:31,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:31,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:31,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/b8b61ecd900e49b2bcd7a2fbb3f5bc92 2023-04-16 20:55:31,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/d86964e8338a4ab0b62cacc2144c3433 2023-04-16 20:55:31,864 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:31,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:31,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/e6712d5e427a47e99c1aeb37f98ae67d 2023-04-16 20:55:31,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:31,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:31,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:31,869 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:31,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:31,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:31,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:31,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,870 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:31,872 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:31,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/b8b61ecd900e49b2bcd7a2fbb3f5bc92 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/b8b61ecd900e49b2bcd7a2fbb3f5bc92 2023-04-16 20:55:31,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:31,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:31,875 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:31,875 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:31,875 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:31,876 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:31,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:31,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:31,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:31,876 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:31,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/b8b61ecd900e49b2bcd7a2fbb3f5bc92, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 20:55:31,878 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/d86964e8338a4ab0b62cacc2144c3433 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d86964e8338a4ab0b62cacc2144c3433 2023-04-16 20:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,878 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,880 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@542ea77f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:31,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:31,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d86964e8338a4ab0b62cacc2144c3433, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 20:55:31,882 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:31,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/e6712d5e427a47e99c1aeb37f98ae67d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e6712d5e427a47e99c1aeb37f98ae67d 2023-04-16 20:55:31,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:31,882 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:31,882 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,882 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:31,889 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:31,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,890 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:31,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:31,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:31,894 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:31,894 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:31,894 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:31,894 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:31,894 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:31,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,896 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:31,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:31,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:31,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e6712d5e427a47e99c1aeb37f98ae67d, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 20:55:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 41b019418b53f9031c0d7e1ee3de5f99 in 489ms, sequenceid=12, compaction requested=false 2023-04-16 20:55:31,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,901 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:31,901 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:31,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:31,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:31,902 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:31,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:31,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:31,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:31,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:31,902 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:31,902 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:31,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:31,903 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,903 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,903 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:31,903 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:31,903 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,903 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:31,903 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,002 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:32,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:32,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:32,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:32,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:32,007 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:32,007 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:32,007 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:32,007 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:32,007 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,007 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,009 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:32,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:32,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:32,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:32,011 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:32,011 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:32,012 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-16 20:55:32,012 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:32,012 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:32,012 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:32,012 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:32,012 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:32,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:32,031 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/c1dbff62588846608f691a80f880f3af 2023-04-16 20:55:32,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:32,052 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:32,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678592064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678592065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,074 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/5a5d01b49d6b452bb426ac6368787042 2023-04-16 20:55:32,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678592066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678592071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678592072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678592173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678592184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678592186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678592388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678592389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678592396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,498 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/2022ff1840504ac093a8f92ae6318eb8 2023-04-16 20:55:32,502 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/c1dbff62588846608f691a80f880f3af as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/c1dbff62588846608f691a80f880f3af 2023-04-16 20:55:32,506 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/c1dbff62588846608f691a80f880f3af, entries=150, sequenceid=45, filesize=11.6 K 2023-04-16 20:55:32,506 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/5a5d01b49d6b452bb426ac6368787042 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a5d01b49d6b452bb426ac6368787042 2023-04-16 20:55:32,510 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a5d01b49d6b452bb426ac6368787042, entries=150, sequenceid=45, filesize=11.6 K 2023-04-16 20:55:32,511 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/2022ff1840504ac093a8f92ae6318eb8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2022ff1840504ac093a8f92ae6318eb8 2023-04-16 20:55:32,514 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2022ff1840504ac093a8f92ae6318eb8, entries=150, sequenceid=45, filesize=11.6 K 2023-04-16 20:55:32,516 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for 41b019418b53f9031c0d7e1ee3de5f99 in 504ms, sequenceid=45, compaction requested=false 2023-04-16 20:55:32,516 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:32,516 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:32,517 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,517 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,520 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0ecc26[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:32,520 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:32,520 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,521 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:32,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:32,523 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:32,527 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:32,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:32,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:32,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:32,527 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:32,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,531 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,533 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:32,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:32,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:32,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:32,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,536 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,536 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,535 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:32,536 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:32,536 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:32,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:32,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:32,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:32,591 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:55:32,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:32,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:32,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:32,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:32,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:32,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:32,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/115c556c65104c6cb3a6091a281eadb1 2023-04-16 20:55:32,636 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:32,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:32,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:32,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:32,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:32,644 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:32,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:32,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:32,644 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:32,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,646 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:32,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:32,646 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:32,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:32,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:32,648 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:32,648 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:32,648 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:32,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:32,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:32,656 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:32,656 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:32,656 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:32,656 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:32,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:32,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:32,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:32,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:32,657 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,658 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,658 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,658 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:32,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e07c3f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:32,659 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:32,659 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,661 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:32,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:32,661 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:32,662 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,662 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:32,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:32,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:32,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,663 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:32,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:32,666 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:32,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:32,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:32,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:32,666 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:32,667 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,667 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:32,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,668 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,670 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678592661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:32,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:32,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:32,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:32,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:32,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:32,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:32,672 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:32,672 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:32,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:32,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678592670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678592693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678592695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678592701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,772 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:32,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:32,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:32,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:32,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:32,775 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:32,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:32,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:32,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:32,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,775 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,777 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,777 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:32,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:32,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,778 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,778 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:32,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678592771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:32,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:32,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:32,779 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:32,779 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:32,779 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:32,779 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:32,781 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,781 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,781 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,781 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:32,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f49cc4[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:32,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:32,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:32,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:32,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:32,784 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,784 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:32,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678592777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,785 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,786 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:32,786 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:32,786 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:32,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:32,793 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:32,794 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:32,794 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:32,794 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:32,794 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:32,798 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,798 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:32,798 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,803 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,804 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,804 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,804 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:32,804 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:32,804 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,804 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,804 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:32,804 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:32,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:32,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:32,804 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:32,805 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:32,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:32,905 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:32,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:32,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:32,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:32,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:32,908 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:32,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:32,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:32,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:32,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,908 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,909 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:32,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:32,910 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,910 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:32,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:32,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:32,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:32,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:32,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:32,914 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,914 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,914 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,914 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:32,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5140c8fa[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:32,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:32,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,916 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:32,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:32,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:32,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:32,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:32,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:32,918 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:32,920 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:32,920 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:32,920 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:32,920 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:32,920 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:32,922 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,922 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:32,922 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:32,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:32,923 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,924 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:32,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:32,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:32,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:32,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:32,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:32,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:32,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:32,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:32,927 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:32,927 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:32,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:32,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678592979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:32,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:32,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678592987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/50687b443cd54d33bd510eca4cde920e 2023-04-16 20:55:33,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/9cd98b19b83440f3b398b94da98a08f1 2023-04-16 20:55:33,028 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:33,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/115c556c65104c6cb3a6091a281eadb1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/115c556c65104c6cb3a6091a281eadb1 2023-04-16 20:55:33,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:33,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:33,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:33,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:33,031 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:33,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:33,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:33,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:33,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:33,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/115c556c65104c6cb3a6091a281eadb1, entries=150, sequenceid=58, filesize=11.6 K 2023-04-16 20:55:33,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:33,033 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:33,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:33,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/50687b443cd54d33bd510eca4cde920e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/50687b443cd54d33bd510eca4cde920e 2023-04-16 20:55:33,033 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,033 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:33,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:33,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:33,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:33,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:33,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:33,036 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:33,036 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:33,036 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:33,036 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:33,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:33,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/50687b443cd54d33bd510eca4cde920e, entries=150, sequenceid=58, filesize=11.6 K 2023-04-16 20:55:33,038 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:33,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:33,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:33,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/9cd98b19b83440f3b398b94da98a08f1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cd98b19b83440f3b398b94da98a08f1 2023-04-16 20:55:33,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:33,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:33,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:33,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:33,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79875108[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:33,040 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:33,040 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:33,042 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:33,042 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,042 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:33,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:33,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:33,042 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:33,042 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:33,042 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:33,043 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cd98b19b83440f3b398b94da98a08f1, entries=150, sequenceid=58, filesize=11.6 K 2023-04-16 20:55:33,043 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 41b019418b53f9031c0d7e1ee3de5f99 in 452ms, sequenceid=58, compaction requested=true 2023-04-16 20:55:33,043 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:33,044 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:33,044 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:33,044 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:33,044 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:33,044 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:33,044 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:33,044 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:33,044 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,044 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:33,045 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:33,045 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:33,045 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:33,045 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/b8b61ecd900e49b2bcd7a2fbb3f5bc92, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/c1dbff62588846608f691a80f880f3af, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/115c556c65104c6cb3a6091a281eadb1] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=34.8 K 2023-04-16 20:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:33,046 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:33,046 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:33,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,046 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:33,046 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d86964e8338a4ab0b62cacc2144c3433, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a5d01b49d6b452bb426ac6368787042, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/50687b443cd54d33bd510eca4cde920e] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=34.8 K 2023-04-16 20:55:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:33,048 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d86964e8338a4ab0b62cacc2144c3433, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681678531404 2023-04-16 20:55:33,048 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting b8b61ecd900e49b2bcd7a2fbb3f5bc92, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681678531404 2023-04-16 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:33,049 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:33,049 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:33,049 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:33,049 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:33,049 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:33,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:33,052 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5a5d01b49d6b452bb426ac6368787042, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681678531444 2023-04-16 20:55:33,052 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c1dbff62588846608f691a80f880f3af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681678531444 2023-04-16 20:55:33,052 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 50687b443cd54d33bd510eca4cde920e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681678532070 2023-04-16 20:55:33,052 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 115c556c65104c6cb3a6091a281eadb1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681678532070 2023-04-16 20:55:33,056 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,056 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:33,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:33,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:33,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:33,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:33,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,060 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:33,060 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:33,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:33,060 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,060 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:33,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:33,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:33,060 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:33,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:33,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:33,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:33,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:33,062 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:33,062 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:33,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:33,065 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:33,069 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:33,107 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/401611c280494aabacc15c48e1800733 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/401611c280494aabacc15c48e1800733 2023-04-16 20:55:33,112 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into 401611c280494aabacc15c48e1800733(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:33,112 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:33,112 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=13, startTime=1681678533044; duration=0sec 2023-04-16 20:55:33,112 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:33,112 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:33,113 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:33,113 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:33,113 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:33,113 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e6712d5e427a47e99c1aeb37f98ae67d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2022ff1840504ac093a8f92ae6318eb8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cd98b19b83440f3b398b94da98a08f1] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=34.8 K 2023-04-16 20:55:33,114 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e6712d5e427a47e99c1aeb37f98ae67d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681678531404 2023-04-16 20:55:33,117 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2022ff1840504ac093a8f92ae6318eb8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681678531444 2023-04-16 20:55:33,117 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9cd98b19b83440f3b398b94da98a08f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681678532070 2023-04-16 20:55:33,132 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:33,162 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:33,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:33,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:33,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:33,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:33,172 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:33,172 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:33,172 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:33,172 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:33,172 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,173 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:33,174 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:33,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:33,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:33,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:33,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,175 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:33,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:33,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:33,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:33,188 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:33,188 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:33,188 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:33,188 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:33,188 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-16 20:55:33,188 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:33,188 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:33,189 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:33,189 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:33,190 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:33,190 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:33,208 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:33,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:33,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678593227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678593230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678593234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678593290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678593299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678593335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678593335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678593341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,508 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/3b12c468e65748a89f39bc4b75c315f2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3b12c468e65748a89f39bc4b75c315f2 2023-04-16 20:55:33,514 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 3b12c468e65748a89f39bc4b75c315f2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:33,514 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:33,514 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=13, startTime=1681678533044; duration=0sec 2023-04-16 20:55:33,514 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:33,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678593542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678593544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678593552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,559 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/0d9f3a77081842c586c28863e9e56fed as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0d9f3a77081842c586c28863e9e56fed 2023-04-16 20:55:33,564 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into 0d9f3a77081842c586c28863e9e56fed(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:33,564 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:33,564 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=13, startTime=1681678533044; duration=0sec 2023-04-16 20:55:33,564 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:33,618 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/89bbd52a006649ec80f048755c9e3c3f 2023-04-16 20:55:33,631 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/1b31c241ae3c437ebe28c275a7dbf038 2023-04-16 20:55:33,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678593799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678593806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678593847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678593852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:33,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:33,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678593858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,041 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/0e04d9dbda5a411d94340af44d54c5e7 2023-04-16 20:55:34,045 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/89bbd52a006649ec80f048755c9e3c3f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/89bbd52a006649ec80f048755c9e3c3f 2023-04-16 20:55:34,057 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/89bbd52a006649ec80f048755c9e3c3f, entries=150, sequenceid=78, filesize=11.6 K 2023-04-16 20:55:34,058 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/1b31c241ae3c437ebe28c275a7dbf038 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/1b31c241ae3c437ebe28c275a7dbf038 2023-04-16 20:55:34,062 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/1b31c241ae3c437ebe28c275a7dbf038, entries=150, sequenceid=78, filesize=11.6 K 2023-04-16 20:55:34,063 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/0e04d9dbda5a411d94340af44d54c5e7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0e04d9dbda5a411d94340af44d54c5e7 2023-04-16 20:55:34,066 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0e04d9dbda5a411d94340af44d54c5e7, entries=150, sequenceid=78, filesize=11.6 K 2023-04-16 20:55:34,067 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 41b019418b53f9031c0d7e1ee3de5f99 in 879ms, sequenceid=78, compaction requested=false 2023-04-16 20:55:34,067 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:34,067 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:34,069 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,069 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,069 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,070 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:34,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738ccb83[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:34,071 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:34,072 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,072 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,073 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:34,073 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:34,073 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:34,073 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,073 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:34,075 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:34,075 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,075 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,075 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:34,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:34,077 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:34,077 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:34,077 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:34,077 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:34,077 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:34,079 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,079 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,079 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:34,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:34,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:34,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,086 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,086 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:34,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:34,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:34,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:34,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:34,086 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:34,086 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,086 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:34,086 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:34,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:34,186 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:34,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:34,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:34,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:34,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:34,189 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:34,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:34,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:34,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:34,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,190 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:34,190 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:34,191 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,191 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:34,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:34,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:34,192 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:34,192 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:34,192 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:34,192 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:34,192 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:34,192 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:34,192 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:34,192 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:55:34,192 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:34,193 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:34,193 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:34,193 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:34,193 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:34,193 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:34,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:34,357 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:34,390 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 20:55:34,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678594413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678594415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678594416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678594518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678594522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678594522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,600 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/1109b549567348f597fba9e48b386498 2023-04-16 20:55:34,611 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/a22b781b482e4b669ba17c33e2cbdee2 2023-04-16 20:55:34,620 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/7f7c7fd06b73486da85359bdfe10d325 2023-04-16 20:55:34,623 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/1109b549567348f597fba9e48b386498 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1109b549567348f597fba9e48b386498 2023-04-16 20:55:34,631 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1109b549567348f597fba9e48b386498, entries=150, sequenceid=93, filesize=11.6 K 2023-04-16 20:55:34,632 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/a22b781b482e4b669ba17c33e2cbdee2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a22b781b482e4b669ba17c33e2cbdee2 2023-04-16 20:55:34,637 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a22b781b482e4b669ba17c33e2cbdee2, entries=150, sequenceid=93, filesize=11.6 K 2023-04-16 20:55:34,638 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/7f7c7fd06b73486da85359bdfe10d325 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7f7c7fd06b73486da85359bdfe10d325 2023-04-16 20:55:34,641 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7f7c7fd06b73486da85359bdfe10d325, entries=150, sequenceid=93, filesize=11.6 K 2023-04-16 20:55:34,641 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 41b019418b53f9031c0d7e1ee3de5f99 in 449ms, sequenceid=93, compaction requested=true 2023-04-16 20:55:34,641 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:34,642 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:34,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:34,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:34,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:34,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:34,644 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,645 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:34,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27ca7b8a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:34,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:34,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,648 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,648 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:34,650 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:34,650 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:34,650 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:34,650 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:34,653 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:34,653 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:34,653 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:34,653 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:34,653 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:34,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:34,655 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,656 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,657 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,662 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,662 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,663 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:34,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:34,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:34,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:34,663 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:34,663 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:34,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:34,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:34,663 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:34,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:34,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:34,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 20:55:34,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:34,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:34,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:34,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:34,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:34,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:34,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/db4a9a672afb48f49d3dee43cc4b8730 2023-04-16 20:55:34,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678594744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678594745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/6696409ff0de458dbb5e332cff34a4f5 2023-04-16 20:55:34,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678594755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,763 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:34,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:34,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:34,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:34,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:34,767 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:34,767 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:34,767 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:34,767 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:34,768 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,768 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:34,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:34,769 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:34,770 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,770 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:34,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:34,771 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:34,771 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:34,771 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:34,771 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:34,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,773 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@709a7091[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:34,776 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:34,776 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,777 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,777 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:34,777 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:34,777 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:34,777 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,777 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:34,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:34,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:34,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:34,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:34,785 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:34,785 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:34,785 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:34,785 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:34,785 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:34,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,789 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,792 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,792 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:34,792 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:34,792 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,793 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,793 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:34,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:34,793 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:34,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:34,793 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,793 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:34,793 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:34,793 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:34,793 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:34,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:34,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678594809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678594811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678594854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678594854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:34,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678594863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,893 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:34,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:34,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:34,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:34,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:34,896 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:34,896 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:34,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:34,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:34,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,898 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:34,898 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:34,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:34,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:34,900 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:34,900 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:34,900 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:34,900 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,901 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:34,901 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:34,903 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,903 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,903 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,903 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3356709b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:34,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:34,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,906 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:34,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:34,907 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:34,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:34,908 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,908 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,908 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:34,908 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:34,911 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:34,911 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:34,911 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:34,911 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:34,911 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:34,912 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,912 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:34,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,912 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:34,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:34,913 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,916 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,920 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,920 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:34,920 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:34,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:34,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:34,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:34,920 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:34,920 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:34,921 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:34,921 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:34,921 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:34,921 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,921 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:34,921 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:34,921 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:34,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:35,021 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:35,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:35,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:35,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:35,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:35,024 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:35,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:35,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:35,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:35,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,032 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:35,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:35,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:35,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:35,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:35,033 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:35,033 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:35,033 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:35,033 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:35,034 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:35,034 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:35,034 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:35,034 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:35,035 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,035 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@261375a1[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:35,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:35,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,038 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:35,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:35,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:35,039 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,039 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:35,041 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:35,043 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:35,043 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:35,043 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:35,043 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:35,043 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:35,045 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,045 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,045 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:35,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:35,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:35,045 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,046 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:35,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,048 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,048 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,048 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,048 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:35,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:35,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:35,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:35,049 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,049 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:35,049 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:35,049 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:35,049 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:35,049 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:35,049 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,049 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:35,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678595059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678595060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678595066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,149 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:35,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:35,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:35,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:35,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:35,152 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:35,152 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:35,152 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:35,152 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:35,153 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,153 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,154 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,154 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:35,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:35,154 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,154 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:35,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:35,155 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:35,155 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:35,155 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:35,155 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:35,155 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:35,158 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,158 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:35,158 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,158 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f15cc2[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:35,159 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:35,159 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,160 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,161 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:35,161 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:35,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:35,161 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:35,162 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,162 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:35,162 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:35,162 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:35,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:35,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:35,165 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:35,165 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:35,165 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:35,165 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:35,165 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:35,166 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,166 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,166 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:35,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:35,167 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,168 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:35,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/34f349a9152a40139e464f867933c153 2023-04-16 20:55:35,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:35,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:35,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:35,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,171 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:35,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:35,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:35,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:35,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:35,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:35,171 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,171 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:35,172 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:35,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/db4a9a672afb48f49d3dee43cc4b8730 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/db4a9a672afb48f49d3dee43cc4b8730 2023-04-16 20:55:35,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/db4a9a672afb48f49d3dee43cc4b8730, entries=200, sequenceid=113, filesize=13.9 K 2023-04-16 20:55:35,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/6696409ff0de458dbb5e332cff34a4f5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6696409ff0de458dbb5e332cff34a4f5 2023-04-16 20:55:35,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6696409ff0de458dbb5e332cff34a4f5, entries=150, sequenceid=113, filesize=11.6 K 2023-04-16 20:55:35,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/34f349a9152a40139e464f867933c153 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/34f349a9152a40139e464f867933c153 2023-04-16 20:55:35,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/34f349a9152a40139e464f867933c153, entries=150, sequenceid=113, filesize=11.6 K 2023-04-16 20:55:35,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 41b019418b53f9031c0d7e1ee3de5f99 in 457ms, sequenceid=113, compaction requested=true 2023-04-16 20:55:35,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:35,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:35,184 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:35,184 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:35,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:35,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:35,185 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:35,185 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49987 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:35,185 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:35,185 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:35,185 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:35,186 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:35,186 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3b12c468e65748a89f39bc4b75c315f2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/1b31c241ae3c437ebe28c275a7dbf038, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a22b781b482e4b669ba17c33e2cbdee2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6696409ff0de458dbb5e332cff34a4f5] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=46.5 K 2023-04-16 20:55:35,186 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/401611c280494aabacc15c48e1800733, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/89bbd52a006649ec80f048755c9e3c3f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1109b549567348f597fba9e48b386498, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/db4a9a672afb48f49d3dee43cc4b8730] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=48.8 K 2023-04-16 20:55:35,186 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 401611c280494aabacc15c48e1800733, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681678532070 2023-04-16 20:55:35,186 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3b12c468e65748a89f39bc4b75c315f2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681678532070 2023-04-16 20:55:35,186 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 89bbd52a006649ec80f048755c9e3c3f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681678532635 2023-04-16 20:55:35,187 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1b31c241ae3c437ebe28c275a7dbf038, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681678532635 2023-04-16 20:55:35,187 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1109b549567348f597fba9e48b386498, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681678533218 2023-04-16 20:55:35,187 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting a22b781b482e4b669ba17c33e2cbdee2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681678533218 2023-04-16 20:55:35,187 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting db4a9a672afb48f49d3dee43cc4b8730, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681678534407 2023-04-16 20:55:35,187 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6696409ff0de458dbb5e332cff34a4f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681678534407 2023-04-16 20:55:35,195 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#431 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:35,195 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:35,209 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/0f0e44038a3b477c89e91d8c32dc71a3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0f0e44038a3b477c89e91d8c32dc71a3 2023-04-16 20:55:35,209 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/577881d3a0c949dfb29fe6a49e2b5420 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/577881d3a0c949dfb29fe6a49e2b5420 2023-04-16 20:55:35,214 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 0f0e44038a3b477c89e91d8c32dc71a3(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:35,214 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:35,214 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=12, startTime=1681678535184; duration=0sec 2023-04-16 20:55:35,214 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:35,214 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:35,216 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:35,216 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:35,216 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:35,216 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0d9f3a77081842c586c28863e9e56fed, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0e04d9dbda5a411d94340af44d54c5e7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7f7c7fd06b73486da85359bdfe10d325, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/34f349a9152a40139e464f867933c153] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=46.5 K 2023-04-16 20:55:35,217 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into 577881d3a0c949dfb29fe6a49e2b5420(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:35,217 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:35,217 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=12, startTime=1681678535184; duration=0sec 2023-04-16 20:55:35,217 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:35,218 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0d9f3a77081842c586c28863e9e56fed, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681678532070 2023-04-16 20:55:35,218 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0e04d9dbda5a411d94340af44d54c5e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681678532635 2023-04-16 20:55:35,218 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7f7c7fd06b73486da85359bdfe10d325, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681678533218 2023-04-16 20:55:35,218 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 34f349a9152a40139e464f867933c153, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681678534407 2023-04-16 20:55:35,225 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#433 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:35,234 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/7d56cb85f5814b3b95098db6a8030108 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7d56cb85f5814b3b95098db6a8030108 2023-04-16 20:55:35,239 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into 7d56cb85f5814b3b95098db6a8030108(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:35,239 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:35,239 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=12, startTime=1681678535184; duration=0sec 2023-04-16 20:55:35,240 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:35,272 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:35,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:35,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:35,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:35,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:35,275 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:35,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:35,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:35,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:35,276 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,276 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:35,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,277 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:35,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:35,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:35,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:35,279 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:35,279 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:35,279 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:55:35,279 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:35,279 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:35,279 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:35,279 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:35,279 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:35,279 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:35,286 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/6feb87ba06354a02b0fdb5b7c62bdda4 2023-04-16 20:55:35,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:35,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:35,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678595394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678595395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678595396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678595498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678595504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678595504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,697 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/c6fe0660b56f4e4a83042a2823d28299 2023-04-16 20:55:35,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678595703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678595712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:35,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:35,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678595712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678596010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678596020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678596021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,110 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/006bb9a03aa4445e822a7c94e59c4018 2023-04-16 20:55:36,114 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/6feb87ba06354a02b0fdb5b7c62bdda4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/6feb87ba06354a02b0fdb5b7c62bdda4 2023-04-16 20:55:36,119 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/6feb87ba06354a02b0fdb5b7c62bdda4, entries=150, sequenceid=128, filesize=11.6 K 2023-04-16 20:55:36,120 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/c6fe0660b56f4e4a83042a2823d28299 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c6fe0660b56f4e4a83042a2823d28299 2023-04-16 20:55:36,124 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c6fe0660b56f4e4a83042a2823d28299, entries=150, sequenceid=128, filesize=11.6 K 2023-04-16 20:55:36,124 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/006bb9a03aa4445e822a7c94e59c4018 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/006bb9a03aa4445e822a7c94e59c4018 2023-04-16 20:55:36,128 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/006bb9a03aa4445e822a7c94e59c4018, entries=150, sequenceid=128, filesize=11.6 K 2023-04-16 20:55:36,129 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 41b019418b53f9031c0d7e1ee3de5f99 in 850ms, sequenceid=128, compaction requested=false 2023-04-16 20:55:36,129 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:36,129 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:36,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:36,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:36,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:36,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:36,131 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:36,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:36,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:36,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:36,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71ccb0f5[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:36,134 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:36,134 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,135 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:36,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:36,135 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:36,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:36,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:36,137 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:36,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:36,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:36,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:36,137 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:36,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:36,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:36,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:36,148 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:36,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,151 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,155 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,155 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:36,155 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:36,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:36,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:36,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:36,155 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,156 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,156 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,156 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,156 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,156 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:36,156 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:36,155 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:36,155 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:36,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:36,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:36,256 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:36,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:36,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:36,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:36,259 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:36,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:36,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:36,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:36,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:36,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:36,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:36,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:36,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:36,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:36,263 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:36,263 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:36,263 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:36,272 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:36,272 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:36,272 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:36,272 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:36,272 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 20:55:36,272 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:36,272 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:36,272 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:36,273 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:36,273 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:36,273 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:36,522 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:36,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678596539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678596539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678596540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678596647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678596647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678596648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,694 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/3be023b035ca479d832fa1e86c215bf6 2023-04-16 20:55:36,742 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/ad7eb2685fef40099ec0fa6b2123d7b0 2023-04-16 20:55:36,776 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/c5effc281862422687af0ffdeefed8e1 2023-04-16 20:55:36,786 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/3be023b035ca479d832fa1e86c215bf6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/3be023b035ca479d832fa1e86c215bf6 2023-04-16 20:55:36,789 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/3be023b035ca479d832fa1e86c215bf6, entries=150, sequenceid=148, filesize=11.8 K 2023-04-16 20:55:36,801 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/ad7eb2685fef40099ec0fa6b2123d7b0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/ad7eb2685fef40099ec0fa6b2123d7b0 2023-04-16 20:55:36,805 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/ad7eb2685fef40099ec0fa6b2123d7b0, entries=150, sequenceid=148, filesize=11.8 K 2023-04-16 20:55:36,806 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/c5effc281862422687af0ffdeefed8e1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/c5effc281862422687af0ffdeefed8e1 2023-04-16 20:55:36,817 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/c5effc281862422687af0ffdeefed8e1, entries=150, sequenceid=148, filesize=11.8 K 2023-04-16 20:55:36,818 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 41b019418b53f9031c0d7e1ee3de5f99 in 546ms, sequenceid=148, compaction requested=true 2023-04-16 20:55:36,818 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:36,818 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:36,818 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:36,818 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:36,818 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:36,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:36,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,824 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:36,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:36,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:36,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@719ec4ff[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:36,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:36,828 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:36,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:36,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:36,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:36,831 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:36,831 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,831 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:36,831 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:36,833 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:36,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:36,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:36,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:36,833 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:36,835 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:36,836 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:36,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:36,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:55:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:36,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:36,841 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,841 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,841 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,841 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:36,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:36,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:36,842 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:36,842 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:36,841 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,842 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,842 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:36,842 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:36,844 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:36,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:36,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:36,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:36,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:36,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/cb764e050f7942a88bc76b8499321b0b 2023-04-16 20:55:36,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/3406f7583cef4be489e6280a8cdfca51 2023-04-16 20:55:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678596887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678596889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678596890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678596890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:36,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678596892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/aa5469d49441424d93ed31e1bc7daf5d 2023-04-16 20:55:36,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/cb764e050f7942a88bc76b8499321b0b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cb764e050f7942a88bc76b8499321b0b 2023-04-16 20:55:36,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cb764e050f7942a88bc76b8499321b0b, entries=200, sequenceid=161, filesize=14.1 K 2023-04-16 20:55:36,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/3406f7583cef4be489e6280a8cdfca51 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3406f7583cef4be489e6280a8cdfca51 2023-04-16 20:55:36,943 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:36,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:36,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:36,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3406f7583cef4be489e6280a8cdfca51, entries=150, sequenceid=161, filesize=11.8 K 2023-04-16 20:55:36,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/aa5469d49441424d93ed31e1bc7daf5d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/aa5469d49441424d93ed31e1bc7daf5d 2023-04-16 20:55:36,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:36,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:36,952 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:36,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:36,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:36,952 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:36,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,954 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:36,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:36,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:36,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:36,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:36,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/aa5469d49441424d93ed31e1bc7daf5d, entries=150, sequenceid=161, filesize=11.8 K 2023-04-16 20:55:36,961 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 41b019418b53f9031c0d7e1ee3de5f99 in 124ms, sequenceid=161, compaction requested=true 2023-04-16 20:55:36,961 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:36,961 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:36,961 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:36,961 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:36,961 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:36,962 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:36,965 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50474 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:36,965 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:36,965 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:36,965 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:36,965 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/577881d3a0c949dfb29fe6a49e2b5420, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/6feb87ba06354a02b0fdb5b7c62bdda4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/3be023b035ca479d832fa1e86c215bf6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cb764e050f7942a88bc76b8499321b0b] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=49.3 K 2023-04-16 20:55:36,966 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:36,966 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:36,966 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0f0e44038a3b477c89e91d8c32dc71a3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c6fe0660b56f4e4a83042a2823d28299, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/ad7eb2685fef40099ec0fa6b2123d7b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3406f7583cef4be489e6280a8cdfca51] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=47.0 K 2023-04-16 20:55:36,966 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 577881d3a0c949dfb29fe6a49e2b5420, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681678534407 2023-04-16 20:55:36,966 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0f0e44038a3b477c89e91d8c32dc71a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681678534407 2023-04-16 20:55:36,966 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6feb87ba06354a02b0fdb5b7c62bdda4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681678534740 2023-04-16 20:55:36,967 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c6fe0660b56f4e4a83042a2823d28299, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681678534740 2023-04-16 20:55:36,967 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3be023b035ca479d832fa1e86c215bf6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681678535389 2023-04-16 20:55:36,967 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ad7eb2685fef40099ec0fa6b2123d7b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681678535389 2023-04-16 20:55:36,968 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3406f7583cef4be489e6280a8cdfca51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681678536532 2023-04-16 20:55:36,968 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting cb764e050f7942a88bc76b8499321b0b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681678536532 2023-04-16 20:55:36,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:36,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:36,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:36,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:36,970 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:36,985 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:36,985 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:36,985 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-16 20:55:36,985 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:36,985 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:36,986 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:36,986 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:36,986 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:36,986 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:37,005 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:37,008 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#444 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:37,013 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:37,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:37,027 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/0729e563738e475a9fb461e7e46eb995 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0729e563738e475a9fb461e7e46eb995 2023-04-16 20:55:37,031 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 0729e563738e475a9fb461e7e46eb995(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:37,032 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:37,032 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=12, startTime=1681678536961; duration=0sec 2023-04-16 20:55:37,032 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:37,032 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:37,033 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:37,033 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:37,033 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:37,033 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7d56cb85f5814b3b95098db6a8030108, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/006bb9a03aa4445e822a7c94e59c4018, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/c5effc281862422687af0ffdeefed8e1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/aa5469d49441424d93ed31e1bc7daf5d] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=47.0 K 2023-04-16 20:55:37,034 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7d56cb85f5814b3b95098db6a8030108, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681678534407 2023-04-16 20:55:37,034 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 006bb9a03aa4445e822a7c94e59c4018, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681678534740 2023-04-16 20:55:37,035 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c5effc281862422687af0ffdeefed8e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681678535389 2023-04-16 20:55:37,035 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting aa5469d49441424d93ed31e1bc7daf5d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681678536532 2023-04-16 20:55:37,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678597029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678597029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678597030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,045 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:37,062 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/51ccc339b80c4926ae30786a51fb2af0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/51ccc339b80c4926ae30786a51fb2af0 2023-04-16 20:55:37,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678597044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678597044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,070 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into 51ccc339b80c4926ae30786a51fb2af0(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:37,070 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:37,070 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=12, startTime=1681678536961; duration=0sec 2023-04-16 20:55:37,070 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:37,111 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/6901af4659c94746b43b66227f27c9a8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6901af4659c94746b43b66227f27c9a8 2023-04-16 20:55:37,127 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into 6901af4659c94746b43b66227f27c9a8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:37,127 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:37,128 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=12, startTime=1681678536962; duration=0sec 2023-04-16 20:55:37,128 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:37,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678597145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678597146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678597147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678597169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678597169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678597361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678597362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678597363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678597372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678597374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,486 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/e0bd6d5fca5e400bbe9ab530daa91f38 2023-04-16 20:55:37,497 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/c43a63cfefe6491f9d4214c406ad6b25 2023-04-16 20:55:37,520 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/acf0dc1aaf8342b3b0c2f3d83d7b6221 2023-04-16 20:55:37,523 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/e0bd6d5fca5e400bbe9ab530daa91f38 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e0bd6d5fca5e400bbe9ab530daa91f38 2023-04-16 20:55:37,528 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e0bd6d5fca5e400bbe9ab530daa91f38, entries=150, sequenceid=180, filesize=11.8 K 2023-04-16 20:55:37,529 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/c43a63cfefe6491f9d4214c406ad6b25 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c43a63cfefe6491f9d4214c406ad6b25 2023-04-16 20:55:37,541 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c43a63cfefe6491f9d4214c406ad6b25, entries=150, sequenceid=180, filesize=11.8 K 2023-04-16 20:55:37,542 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/acf0dc1aaf8342b3b0c2f3d83d7b6221 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/acf0dc1aaf8342b3b0c2f3d83d7b6221 2023-04-16 20:55:37,561 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/acf0dc1aaf8342b3b0c2f3d83d7b6221, entries=150, sequenceid=180, filesize=11.8 K 2023-04-16 20:55:37,562 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 41b019418b53f9031c0d7e1ee3de5f99 in 577ms, sequenceid=180, compaction requested=false 2023-04-16 20:55:37,562 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:37,562 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:37,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c1d1910[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:37,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:37,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,567 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,568 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,568 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:37,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:37,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:37,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:37,571 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:37,571 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:37,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:37,575 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:37,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:37,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:37,575 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:37,575 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:37,576 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,576 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:37,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,576 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:37,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:37,577 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,579 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,583 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:37,583 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:37,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:37,583 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,583 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,583 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,583 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,583 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,583 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,584 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:37,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:37,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:37,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:37,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:37,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:37,585 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:37,585 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:37,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:37,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 20:55:37,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:37,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:37,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:37,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:37,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:37,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:37,685 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:37,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:37,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:37,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:37,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:37,695 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:37,695 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:37,695 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:37,695 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:37,696 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,697 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,698 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:37,698 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:37,698 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,698 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:37,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678597698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678597698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678597700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678597700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678597701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:37,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:37,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:37,774 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:37,774 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:37,774 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:37,774 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:37,779 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:37,779 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:37,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:37,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14d25b1d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:37,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:37,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,782 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:37,782 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:37,782 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:37,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:37,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:37,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:37,784 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,785 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:37,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:37,786 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:37,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:37,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:37,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:37,786 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:37,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:37,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:37,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:37,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:37,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:37,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,792 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:37,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,799 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:37,799 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:37,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:37,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:37,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:37,805 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,805 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,805 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:37,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:37,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:37,806 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:37,806 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:37,805 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,806 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,806 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,806 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,806 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678597811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678597811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678597811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678597812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:37,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678597813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,906 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:37,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:37,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:37,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:37,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:37,909 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:37,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:37,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:37,910 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:37,910 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,910 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,913 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:37,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,914 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,917 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-16 20:55:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:37,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:37,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:37,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:37,934 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:37,935 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:37,935 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:37,941 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:37,941 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:37,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:37,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:37,950 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,950 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,953 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,953 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:37,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:37,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d0a6a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:37,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:37,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,957 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:37,959 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/b8b61ecd900e49b2bcd7a2fbb3f5bc92, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/c1dbff62588846608f691a80f880f3af, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/401611c280494aabacc15c48e1800733, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/115c556c65104c6cb3a6091a281eadb1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/89bbd52a006649ec80f048755c9e3c3f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1109b549567348f597fba9e48b386498, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/db4a9a672afb48f49d3dee43cc4b8730, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/577881d3a0c949dfb29fe6a49e2b5420, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/6feb87ba06354a02b0fdb5b7c62bdda4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/3be023b035ca479d832fa1e86c215bf6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cb764e050f7942a88bc76b8499321b0b] to archive 2023-04-16 20:55:37,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:37,960 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:37,962 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/b8b61ecd900e49b2bcd7a2fbb3f5bc92 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/b8b61ecd900e49b2bcd7a2fbb3f5bc92 2023-04-16 20:55:37,963 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/c1dbff62588846608f691a80f880f3af to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/c1dbff62588846608f691a80f880f3af 2023-04-16 20:55:37,968 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/401611c280494aabacc15c48e1800733 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/401611c280494aabacc15c48e1800733 2023-04-16 20:55:37,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,969 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:37,969 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:37,969 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:37,985 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:37,985 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:37,986 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:37,986 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:37,986 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:37,987 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/115c556c65104c6cb3a6091a281eadb1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/115c556c65104c6cb3a6091a281eadb1 2023-04-16 20:55:37,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:37,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,988 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:37,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,990 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/89bbd52a006649ec80f048755c9e3c3f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/89bbd52a006649ec80f048755c9e3c3f 2023-04-16 20:55:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,990 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:37,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:37,992 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:37,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,992 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1109b549567348f597fba9e48b386498 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1109b549567348f597fba9e48b386498 2023-04-16 20:55:37,994 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,994 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,994 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:37,994 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:37,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:37,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:37,994 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:37,994 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:37,994 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:37,994 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:37,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:37,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:37,994 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:37,994 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,995 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:37,995 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:37,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:37,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:37,995 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/db4a9a672afb48f49d3dee43cc4b8730 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/db4a9a672afb48f49d3dee43cc4b8730 2023-04-16 20:55:37,998 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/577881d3a0c949dfb29fe6a49e2b5420 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/577881d3a0c949dfb29fe6a49e2b5420 2023-04-16 20:55:38,001 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/6feb87ba06354a02b0fdb5b7c62bdda4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/6feb87ba06354a02b0fdb5b7c62bdda4 2023-04-16 20:55:38,002 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/3be023b035ca479d832fa1e86c215bf6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/3be023b035ca479d832fa1e86c215bf6 2023-04-16 20:55:38,003 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cb764e050f7942a88bc76b8499321b0b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cb764e050f7942a88bc76b8499321b0b 2023-04-16 20:55:38,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678598025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,048 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d86964e8338a4ab0b62cacc2144c3433, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a5d01b49d6b452bb426ac6368787042, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3b12c468e65748a89f39bc4b75c315f2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/50687b443cd54d33bd510eca4cde920e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/1b31c241ae3c437ebe28c275a7dbf038, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a22b781b482e4b669ba17c33e2cbdee2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0f0e44038a3b477c89e91d8c32dc71a3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6696409ff0de458dbb5e332cff34a4f5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c6fe0660b56f4e4a83042a2823d28299, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/ad7eb2685fef40099ec0fa6b2123d7b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3406f7583cef4be489e6280a8cdfca51] to archive 2023-04-16 20:55:38,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678598025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678598025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,049 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:38,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678598026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678598026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,055 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d86964e8338a4ab0b62cacc2144c3433 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d86964e8338a4ab0b62cacc2144c3433 2023-04-16 20:55:38,056 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a5d01b49d6b452bb426ac6368787042 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a5d01b49d6b452bb426ac6368787042 2023-04-16 20:55:38,057 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3b12c468e65748a89f39bc4b75c315f2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3b12c468e65748a89f39bc4b75c315f2 2023-04-16 20:55:38,058 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/50687b443cd54d33bd510eca4cde920e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/50687b443cd54d33bd510eca4cde920e 2023-04-16 20:55:38,061 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/1b31c241ae3c437ebe28c275a7dbf038 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/1b31c241ae3c437ebe28c275a7dbf038 2023-04-16 20:55:38,062 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a22b781b482e4b669ba17c33e2cbdee2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a22b781b482e4b669ba17c33e2cbdee2 2023-04-16 20:55:38,063 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0f0e44038a3b477c89e91d8c32dc71a3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0f0e44038a3b477c89e91d8c32dc71a3 2023-04-16 20:55:38,065 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6696409ff0de458dbb5e332cff34a4f5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6696409ff0de458dbb5e332cff34a4f5 2023-04-16 20:55:38,066 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c6fe0660b56f4e4a83042a2823d28299 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c6fe0660b56f4e4a83042a2823d28299 2023-04-16 20:55:38,067 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/ad7eb2685fef40099ec0fa6b2123d7b0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/ad7eb2685fef40099ec0fa6b2123d7b0 2023-04-16 20:55:38,073 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3406f7583cef4be489e6280a8cdfca51 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/3406f7583cef4be489e6280a8cdfca51 2023-04-16 20:55:38,095 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:38,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:38,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:38,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:38,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:38,108 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:38,108 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,108 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:38,108 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:38,108 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,108 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,111 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:38,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:38,111 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,111 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,111 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:38,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:38,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:38,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:38,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:38,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,139 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:38,151 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,151 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:38,151 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e6712d5e427a47e99c1aeb37f98ae67d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2022ff1840504ac093a8f92ae6318eb8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0d9f3a77081842c586c28863e9e56fed, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cd98b19b83440f3b398b94da98a08f1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0e04d9dbda5a411d94340af44d54c5e7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7f7c7fd06b73486da85359bdfe10d325, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7d56cb85f5814b3b95098db6a8030108, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/34f349a9152a40139e464f867933c153, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/006bb9a03aa4445e822a7c94e59c4018, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/c5effc281862422687af0ffdeefed8e1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/aa5469d49441424d93ed31e1bc7daf5d] to archive 2023-04-16 20:55:38,151 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:38,152 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:38,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:38,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:38,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:38,153 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:38,155 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e6712d5e427a47e99c1aeb37f98ae67d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e6712d5e427a47e99c1aeb37f98ae67d 2023-04-16 20:55:38,156 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2022ff1840504ac093a8f92ae6318eb8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2022ff1840504ac093a8f92ae6318eb8 2023-04-16 20:55:38,157 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0d9f3a77081842c586c28863e9e56fed to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0d9f3a77081842c586c28863e9e56fed 2023-04-16 20:55:38,158 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,158 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,159 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cd98b19b83440f3b398b94da98a08f1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cd98b19b83440f3b398b94da98a08f1 2023-04-16 20:55:38,159 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,159 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:38,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,160 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0e04d9dbda5a411d94340af44d54c5e7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0e04d9dbda5a411d94340af44d54c5e7 2023-04-16 20:55:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,161 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7f7c7fd06b73486da85359bdfe10d325 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7f7c7fd06b73486da85359bdfe10d325 2023-04-16 20:55:38,162 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7d56cb85f5814b3b95098db6a8030108 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7d56cb85f5814b3b95098db6a8030108 2023-04-16 20:55:38,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:38,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7338fb8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:38,163 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:38,163 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,163 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/34f349a9152a40139e464f867933c153 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/34f349a9152a40139e464f867933c153 2023-04-16 20:55:38,165 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/006bb9a03aa4445e822a7c94e59c4018 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/006bb9a03aa4445e822a7c94e59c4018 2023-04-16 20:55:38,165 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,165 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,165 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:38,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:38,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:38,165 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:38,166 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/c5effc281862422687af0ffdeefed8e1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/c5effc281862422687af0ffdeefed8e1 2023-04-16 20:55:38,167 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:38,167 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:38,167 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:38,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,167 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/e192d9baa24742f989acf118a793bd7a 2023-04-16 20:55:38,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,169 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/aa5469d49441424d93ed31e1bc7daf5d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/aa5469d49441424d93ed31e1bc7daf5d 2023-04-16 20:55:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:38,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:38,170 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:38,170 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:38,170 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:38,170 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:38,170 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:38,175 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,175 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:38,175 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,176 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,178 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:38,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:38,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:38,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:38,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:38,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:38,181 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,182 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:38,182 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:38,182 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:38,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:38,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/e7d9ca646ecf42b682482cc62445dc4a 2023-04-16 20:55:38,283 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:38,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:38,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:38,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:38,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:38,312 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:38,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:38,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:38,313 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,313 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,314 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,314 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,314 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:38,314 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:38,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:38,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:38,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:38,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:38,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,319 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:38,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:38,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:38,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:38,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:38,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:38,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:38,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:38,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:38,340 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,342 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:38,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c2c3d7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:38,347 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:38,347 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,349 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:38,349 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:38,349 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:38,349 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:38,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,352 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:38,352 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:38,352 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:38,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:38,356 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:38,356 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:38,356 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:38,356 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:38,356 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:38,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678598350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678598350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,358 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,358 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:38,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,358 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:38,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:38,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678598352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678598353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:38,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:38,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:38,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:38,366 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:38,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:38,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:38,367 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:38,367 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:38,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:38,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:38,366 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678598358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,467 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:38,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:38,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:38,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:38,488 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:38,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:38,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:38,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,490 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:38,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:38,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:38,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,491 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:38,491 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,491 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:38,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:38,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:38,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:38,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:38,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:38,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:38,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:38,524 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,524 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:38,524 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:38,524 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:38,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:38,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:38,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:38,532 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,533 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,533 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:38,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254e7e01[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:38,536 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:38,536 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,537 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,537 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:38,537 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:38,537 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:38,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,537 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:38,538 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:38,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:38,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,539 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:38,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:38,541 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:38,541 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:38,541 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:38,541 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:38,541 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:38,548 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,548 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:38,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,548 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,550 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,555 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:38,555 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:38,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:38,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,556 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:38,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:38,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:38,556 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:38,556 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:38,556 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:38,657 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:38,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:38,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:38,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:38,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:38,668 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:38,668 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,668 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:38,668 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:38,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,670 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:38,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:38,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:38,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:38,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:38,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:38,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:38,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:38,691 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:38,691 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:38,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/f6f0d5c1f9894834b547ef22a616a687 2023-04-16 20:55:38,695 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,695 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:38,695 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:38,695 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:38,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:38,695 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:38,710 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:38,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/e192d9baa24742f989acf118a793bd7a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e192d9baa24742f989acf118a793bd7a 2023-04-16 20:55:38,712 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,712 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,713 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,713 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:38,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:38,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df8ee98[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:38,715 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:38,715 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,717 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:38,717 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:38,718 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:38,718 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,718 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:38,719 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,719 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,719 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:38,719 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:38,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:38,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:38,722 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:38,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:38,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:38,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:38,722 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:38,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:38,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:38,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:38,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:38,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,724 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,726 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:38,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:38,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:38,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:38,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,728 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:38,729 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:38,728 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:38,729 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:38,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:38,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:38,729 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,729 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:38,729 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:38,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e192d9baa24742f989acf118a793bd7a, entries=200, sequenceid=197, filesize=14.1 K 2023-04-16 20:55:38,738 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/e7d9ca646ecf42b682482cc62445dc4a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e7d9ca646ecf42b682482cc62445dc4a 2023-04-16 20:55:38,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e7d9ca646ecf42b682482cc62445dc4a, entries=150, sequenceid=197, filesize=11.8 K 2023-04-16 20:55:38,743 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/f6f0d5c1f9894834b547ef22a616a687 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/f6f0d5c1f9894834b547ef22a616a687 2023-04-16 20:55:38,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/f6f0d5c1f9894834b547ef22a616a687, entries=150, sequenceid=197, filesize=11.8 K 2023-04-16 20:55:38,747 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 41b019418b53f9031c0d7e1ee3de5f99 in 1075ms, sequenceid=197, compaction requested=true 2023-04-16 20:55:38,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:38,747 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:38,748 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:38,748 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:38,748 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:38,749 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38874 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:38,749 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:38,749 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:38,749 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,749 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/51ccc339b80c4926ae30786a51fb2af0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e0bd6d5fca5e400bbe9ab530daa91f38, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e192d9baa24742f989acf118a793bd7a] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=38.0 K 2023-04-16 20:55:38,749 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:38,749 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:38,749 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 51ccc339b80c4926ae30786a51fb2af0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681678536532 2023-04-16 20:55:38,749 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,750 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0729e563738e475a9fb461e7e46eb995, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c43a63cfefe6491f9d4214c406ad6b25, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e7d9ca646ecf42b682482cc62445dc4a] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=35.6 K 2023-04-16 20:55:38,750 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e0bd6d5fca5e400bbe9ab530daa91f38, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681678536886 2023-04-16 20:55:38,750 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0729e563738e475a9fb461e7e46eb995, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681678536532 2023-04-16 20:55:38,750 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e192d9baa24742f989acf118a793bd7a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681678537027 2023-04-16 20:55:38,750 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c43a63cfefe6491f9d4214c406ad6b25, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681678536886 2023-04-16 20:55:38,751 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e7d9ca646ecf42b682482cc62445dc4a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681678537027 2023-04-16 20:55:38,766 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#452 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:38,774 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#453 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:38,795 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/b613bbbe20c34034ae2189151b571211 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/b613bbbe20c34034ae2189151b571211 2023-04-16 20:55:38,800 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into b613bbbe20c34034ae2189151b571211(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:38,800 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:38,800 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=13, startTime=1681678538748; duration=0sec 2023-04-16 20:55:38,800 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:38,800 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:38,801 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:38,801 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:38,801 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,801 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6901af4659c94746b43b66227f27c9a8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/acf0dc1aaf8342b3b0c2f3d83d7b6221, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/f6f0d5c1f9894834b547ef22a616a687] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=35.6 K 2023-04-16 20:55:38,803 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6901af4659c94746b43b66227f27c9a8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681678536532 2023-04-16 20:55:38,803 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting acf0dc1aaf8342b3b0c2f3d83d7b6221, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681678536886 2023-04-16 20:55:38,803 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting f6f0d5c1f9894834b547ef22a616a687, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681678537027 2023-04-16 20:55:38,810 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#454 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:38,830 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:38,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:38,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:38,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:38,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:38,837 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:38,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:38,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:38,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:38,840 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,841 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,841 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:38,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:38,843 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/7de75c849671497d8277bbd8f8e1a947 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7de75c849671497d8277bbd8f8e1a947 2023-04-16 20:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:38,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:38,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:38,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:38,848 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into 7de75c849671497d8277bbd8f8e1a947(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:38,848 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:38,848 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=13, startTime=1681678538749; duration=0sec 2023-04-16 20:55:38,848 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:38,849 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:38,850 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:38,850 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-16 20:55:38,850 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:38,850 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:38,850 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:38,850 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:38,850 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:38,850 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:38,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:38,865 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:38,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678598876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678598877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678598884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678598885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678598887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678598988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678598988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678598989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:38,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:38,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678598995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678599002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,198 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/5c9d56f9d219470eb66bd6f4dbb6060d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/5c9d56f9d219470eb66bd6f4dbb6060d 2023-04-16 20:55:39,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678599197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678599196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678599197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,204 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into 5c9d56f9d219470eb66bd6f4dbb6060d(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:39,204 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:39,204 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=13, startTime=1681678538747; duration=0sec 2023-04-16 20:55:39,204 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:39,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678599200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678599209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,281 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/effe7881e062402789789e85f4e6731b 2023-04-16 20:55:39,294 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/907499b4e85b461fa24d428d9bcb0259 2023-04-16 20:55:39,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678599503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678599505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678599507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678599510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:39,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678599518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,710 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/9f25557b666e4666afd2a553aeb2abc5 2023-04-16 20:55:39,716 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/effe7881e062402789789e85f4e6731b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/effe7881e062402789789e85f4e6731b 2023-04-16 20:55:39,720 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/effe7881e062402789789e85f4e6731b, entries=150, sequenceid=217, filesize=11.8 K 2023-04-16 20:55:39,722 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/907499b4e85b461fa24d428d9bcb0259 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/907499b4e85b461fa24d428d9bcb0259 2023-04-16 20:55:39,725 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/907499b4e85b461fa24d428d9bcb0259, entries=150, sequenceid=217, filesize=11.8 K 2023-04-16 20:55:39,726 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/9f25557b666e4666afd2a553aeb2abc5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9f25557b666e4666afd2a553aeb2abc5 2023-04-16 20:55:39,745 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9f25557b666e4666afd2a553aeb2abc5, entries=150, sequenceid=217, filesize=11.8 K 2023-04-16 20:55:39,746 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 41b019418b53f9031c0d7e1ee3de5f99 in 896ms, sequenceid=217, compaction requested=false 2023-04-16 20:55:39,746 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:39,746 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:39,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:39,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:39,749 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:39,750 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:39,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:39,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@562df6d0[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:39,750 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:39,750 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:39,751 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:39,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:39,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:39,753 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:39,753 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,753 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:39,753 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:39,756 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:39,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:39,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:39,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:39,756 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:39,758 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:39,758 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:39,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:39,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:39,759 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:39,761 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,762 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,767 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,767 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:39,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:39,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:39,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:39,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:39,768 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:39,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:39,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:39,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:39,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:39,768 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:39,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:39,768 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:39,768 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:39,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:39,868 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:39,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:39,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:39,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:39,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:39,872 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:39,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:39,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:39,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:39,873 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:39,873 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:39,874 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:39,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:39,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:39,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:39,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:39,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:39,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:39,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:39,888 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:39,888 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:39,888 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:39,889 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:39,889 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:39,889 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:39,889 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:39,889 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 20:55:39,889 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:39,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:39,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:39,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:39,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:39,890 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:39,939 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/f7ad5fbd640a4319b0984a2d108618f7 2023-04-16 20:55:40,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:40,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:40,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678600033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678600040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678600041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678600042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678600043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678600145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678600148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678600148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678600154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678600154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678600350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678600351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678600352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678600361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678600361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,374 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/84ea2eb67247476aad22d4db08df50c8 2023-04-16 20:55:40,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678600659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678600659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678600660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678600667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:40,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36054 deadline: 1681678600669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,784 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/ddf9a7b53add4a50863d90fc894d96f6 2023-04-16 20:55:40,789 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/f7ad5fbd640a4319b0984a2d108618f7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/f7ad5fbd640a4319b0984a2d108618f7 2023-04-16 20:55:40,792 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/f7ad5fbd640a4319b0984a2d108618f7, entries=150, sequenceid=232, filesize=11.8 K 2023-04-16 20:55:40,793 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/84ea2eb67247476aad22d4db08df50c8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/84ea2eb67247476aad22d4db08df50c8 2023-04-16 20:55:40,796 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/84ea2eb67247476aad22d4db08df50c8, entries=150, sequenceid=232, filesize=11.8 K 2023-04-16 20:55:40,797 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/ddf9a7b53add4a50863d90fc894d96f6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ddf9a7b53add4a50863d90fc894d96f6 2023-04-16 20:55:40,800 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ddf9a7b53add4a50863d90fc894d96f6, entries=150, sequenceid=232, filesize=11.8 K 2023-04-16 20:55:40,801 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 41b019418b53f9031c0d7e1ee3de5f99 in 912ms, sequenceid=232, compaction requested=true 2023-04-16 20:55:40,801 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:40,801 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:40,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:40,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:40,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:40,801 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:40,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:40,803 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:40,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:40,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:40,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:40,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63c45bb9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:40,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:40,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:40,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:40,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:40,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:40,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:40,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:40,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:40,813 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,813 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:40,814 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,814 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:40,814 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:40,814 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:40,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:40,817 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:40,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:40,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:40,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:40,817 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:40,820 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:40,820 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:40,820 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:40,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:40,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:40,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:40,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:40,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:40,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:40,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:40,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:40,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:40,821 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,823 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:40,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:40,829 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:40,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:40,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:40,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:40,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:40,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:40,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:40,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:40,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:40,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:40,830 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:40,830 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:40,930 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:40,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:40,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:40,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:40,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:40,936 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:40,936 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:40,936 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:40,936 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:40,937 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:40,937 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:40,938 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:40,938 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:40,938 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:40,938 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:40,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:40,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:40,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:40,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:40,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:40,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:40,948 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:40,949 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:40,950 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-16 20:55:40,950 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:40,950 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:40,950 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:40,950 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:40,950 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:40,950 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:40,966 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/9c1a3b69b0d94ced88a2c7afd4e0862f 2023-04-16 20:55:41,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:41,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:41,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678601180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678601186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678601187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678601188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678601188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678601289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678601295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678601296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678601296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678601296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,384 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/02e0d2c2ed0e41eda3b62932614e5087 2023-04-16 20:55:41,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678601496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678601500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678601503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678601503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36096 deadline: 1681678601504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,801 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/4e1c770dde8a4e4082b42903a2aa680d 2023-04-16 20:55:41,805 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/9c1a3b69b0d94ced88a2c7afd4e0862f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/9c1a3b69b0d94ced88a2c7afd4e0862f 2023-04-16 20:55:41,809 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/9c1a3b69b0d94ced88a2c7afd4e0862f, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 20:55:41,810 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/02e0d2c2ed0e41eda3b62932614e5087 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/02e0d2c2ed0e41eda3b62932614e5087 2023-04-16 20:55:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678601804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678601804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,813 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/02e0d2c2ed0e41eda3b62932614e5087, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 20:55:41,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681678601807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,814 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/4e1c770dde8a4e4082b42903a2aa680d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4e1c770dde8a4e4082b42903a2aa680d 2023-04-16 20:55:41,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678601809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,818 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4e1c770dde8a4e4082b42903a2aa680d, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 20:55:41,819 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 41b019418b53f9031c0d7e1ee3de5f99 in 869ms, sequenceid=250, compaction requested=true 2023-04-16 20:55:41,819 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:41,819 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:41,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:41,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:41,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:41,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:41,822 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,822 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ac9fb6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:41,825 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:41,825 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:41,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-16 20:55:41,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:41,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:41,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:41,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:41,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:41,827 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:41,827 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:41,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:41,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:41,830 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:41,830 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:41,830 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:41,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:41,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:41,830 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:41,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:41,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:41,830 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:41,830 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:41,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,832 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:41,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,833 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:41,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:41,835 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:41,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:41,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:41,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:41,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:41,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:41,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:41,838 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:41,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:41,839 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:41,839 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:41,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:41,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/1d0d9b47260d4046981e79a3f9f99fa5 2023-04-16 20:55:41,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:41,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678601914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,939 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:41,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:41,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:41,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:41,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:41,943 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:41,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:41,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:41,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:41,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:41,945 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:41,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:41,945 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:41,945 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,945 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:41,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:41,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:41,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:41,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:41,946 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:41,946 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:41,946 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:41,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:41,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:41,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:41,947 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:41,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:41,950 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,950 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:41,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:41,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f2a2bb5[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:41,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:41,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,955 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:41,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:41,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:41,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:41,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:41,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:41,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:41,958 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:41,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:41,960 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:41,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:41,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:41,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:41,960 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:41,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:41,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:41,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:41,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:41,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:41,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:41,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,974 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:41,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:41,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:41,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:41,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:41,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:41,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:41,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:41,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:41,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,980 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:41,980 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:41,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:41,980 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:41,980 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:41,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678602026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,080 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:42,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:42,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:42,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:42,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:42,084 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:42,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:42,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:42,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,091 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,091 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:42,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:42,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:42,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:42,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:42,103 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,103 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:42,103 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:42,103 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:42,104 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,104 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:42,104 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:42,106 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,107 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,108 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,108 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:42,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:42,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688d1f6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:42,109 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:42,109 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,110 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,110 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:42,110 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:42,110 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:42,110 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,110 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:42,113 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,113 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:42,113 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,113 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:42,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:42,120 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:42,120 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:42,120 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:42,120 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:42,120 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:42,122 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,122 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,122 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,125 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:42,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:42,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:42,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,130 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:42,130 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:42,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,230 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:42,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:42,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:42,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:42,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:42,235 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:42,235 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,235 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:42,235 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:42,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,239 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,239 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,239 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:42,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:42,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:42,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:42,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:42,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678602235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:42,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:42,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:42,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:42,248 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:42,250 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,251 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:42,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:42,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47d04b0d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:42,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:42,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,253 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:42,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:42,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:42,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:42,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:42,259 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:42,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:42,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:42,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:42,259 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:42,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/939caa7e695b40119520779da633dae6 2023-04-16 20:55:42,265 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,265 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,265 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,267 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,268 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,272 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:42,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:42,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:42,278 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:42,278 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:42,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/8bea3d891d994f47bfdd5f056bbb7c31 2023-04-16 20:55:42,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/1d0d9b47260d4046981e79a3f9f99fa5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1d0d9b47260d4046981e79a3f9f99fa5 2023-04-16 20:55:42,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1d0d9b47260d4046981e79a3f9f99fa5, entries=200, sequenceid=265, filesize=14.3 K 2023-04-16 20:55:42,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/939caa7e695b40119520779da633dae6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/939caa7e695b40119520779da633dae6 2023-04-16 20:55:42,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/939caa7e695b40119520779da633dae6, entries=150, sequenceid=265, filesize=11.9 K 2023-04-16 20:55:42,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/8bea3d891d994f47bfdd5f056bbb7c31 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/8bea3d891d994f47bfdd5f056bbb7c31 2023-04-16 20:55:42,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/8bea3d891d994f47bfdd5f056bbb7c31, entries=150, sequenceid=265, filesize=11.9 K 2023-04-16 20:55:42,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 41b019418b53f9031c0d7e1ee3de5f99 in 475ms, sequenceid=265, compaction requested=true 2023-04-16 20:55:42,301 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:42,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:42,302 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:55:42,302 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:55:42,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:42,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:42,303 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62873 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:55:42,303 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60433 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:55:42,304 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:42,304 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:42,304 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,304 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,304 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/5c9d56f9d219470eb66bd6f4dbb6060d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/effe7881e062402789789e85f4e6731b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/f7ad5fbd640a4319b0984a2d108618f7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/9c1a3b69b0d94ced88a2c7afd4e0862f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1d0d9b47260d4046981e79a3f9f99fa5] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=61.4 K 2023-04-16 20:55:42,304 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/b613bbbe20c34034ae2189151b571211, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/907499b4e85b461fa24d428d9bcb0259, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/84ea2eb67247476aad22d4db08df50c8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/02e0d2c2ed0e41eda3b62932614e5087, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/939caa7e695b40119520779da633dae6] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=59.0 K 2023-04-16 20:55:42,306 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b613bbbe20c34034ae2189151b571211, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681678537027 2023-04-16 20:55:42,306 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 5c9d56f9d219470eb66bd6f4dbb6060d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681678537027 2023-04-16 20:55:42,306 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 907499b4e85b461fa24d428d9bcb0259, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1681678537696 2023-04-16 20:55:42,306 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting effe7881e062402789789e85f4e6731b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1681678537696 2023-04-16 20:55:42,307 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 84ea2eb67247476aad22d4db08df50c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681678538882 2023-04-16 20:55:42,307 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting f7ad5fbd640a4319b0984a2d108618f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681678538882 2023-04-16 20:55:42,307 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 02e0d2c2ed0e41eda3b62932614e5087, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681678540031 2023-04-16 20:55:42,307 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 9c1a3b69b0d94ced88a2c7afd4e0862f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681678540031 2023-04-16 20:55:42,307 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 939caa7e695b40119520779da633dae6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678541186 2023-04-16 20:55:42,308 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1d0d9b47260d4046981e79a3f9f99fa5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678541186 2023-04-16 20:55:42,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:42,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-16 20:55:42,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:42,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:42,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:42,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:42,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:42,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:42,336 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#467 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:42,342 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#469 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:42,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678602344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678602345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678602353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,368 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/01c65d6e4fea4a34b0a0d3cd1824cf50 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/01c65d6e4fea4a34b0a0d3cd1824cf50 2023-04-16 20:55:42,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678602355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,371 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 01c65d6e4fea4a34b0a0d3cd1824cf50(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:42,372 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:42,372 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=11, startTime=1681678542302; duration=0sec 2023-04-16 20:55:42,372 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:42,372 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:55:42,373 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60433 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:55:42,373 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:42,373 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,373 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7de75c849671497d8277bbd8f8e1a947, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9f25557b666e4666afd2a553aeb2abc5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ddf9a7b53add4a50863d90fc894d96f6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4e1c770dde8a4e4082b42903a2aa680d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/8bea3d891d994f47bfdd5f056bbb7c31] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=59.0 K 2023-04-16 20:55:42,375 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7de75c849671497d8277bbd8f8e1a947, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681678537027 2023-04-16 20:55:42,376 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9f25557b666e4666afd2a553aeb2abc5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1681678537696 2023-04-16 20:55:42,376 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ddf9a7b53add4a50863d90fc894d96f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681678538882 2023-04-16 20:55:42,376 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 4e1c770dde8a4e4082b42903a2aa680d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681678540031 2023-04-16 20:55:42,377 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8bea3d891d994f47bfdd5f056bbb7c31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678541186 2023-04-16 20:55:42,378 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:42,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:42,380 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/196c679e36d74459b65e6ca8e5399a82 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/196c679e36d74459b65e6ca8e5399a82 2023-04-16 20:55:42,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:42,385 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into 196c679e36d74459b65e6ca8e5399a82(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:42,385 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:42,385 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=11, startTime=1681678542302; duration=0sec 2023-04-16 20:55:42,386 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:42,398 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#470 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:42,404 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:42,404 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,404 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:42,404 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:42,405 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,405 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,407 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,407 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,407 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:42,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:42,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:42,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:42,412 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:42,412 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,412 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:42,416 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,416 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:42,416 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:42,416 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:42,418 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,418 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,418 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,418 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:42,421 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/e118f4ec2f0e458eb016f161d2024508 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e118f4ec2f0e458eb016f161d2024508 2023-04-16 20:55:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:42,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16502d6e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:42,423 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:42,424 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,426 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,426 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:42,426 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:42,426 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:42,426 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,426 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:42,427 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into e118f4ec2f0e458eb016f161d2024508(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:42,427 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:42,427 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=11, startTime=1681678542302; duration=0sec 2023-04-16 20:55:42,427 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:42,428 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,428 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:42,428 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,428 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:42,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:42,433 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:42,433 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:42,433 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:42,433 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:42,433 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:42,435 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,435 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,435 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,435 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,436 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,438 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,438 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,438 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,438 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,439 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:42,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:42,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,439 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:42,439 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:42,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:42,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678602456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678602456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678602456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678602471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,539 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:42,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:42,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:42,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:42,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:42,543 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:42,543 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,543 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:42,543 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:42,543 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,543 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,545 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,546 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,546 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,546 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:42,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:42,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:42,547 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:42,547 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,547 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:42,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:42,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:42,548 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,548 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:42,548 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:42,548 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:42,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:42,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:42,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:42,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678602544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,550 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:42,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c9db92[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:42,552 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:42,552 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,553 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,554 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:42,554 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:42,554 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:42,554 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,554 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:42,555 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,555 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:42,555 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,555 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:42,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:42,557 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:42,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:42,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:42,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:42,558 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:42,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,560 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,563 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:42,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:42,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,564 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:42,564 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:42,564 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:42,564 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678602661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,664 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:42,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678602662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678602662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:42,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:42,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:42,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:42,668 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:42,668 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,668 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:42,668 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:42,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,670 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:42,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:42,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:42,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:42,673 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,673 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:42,673 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:42,673 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:42,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:42,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:42,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:42,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678602679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,683 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,683 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:42,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:42,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@380ea21c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:42,685 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:42,685 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,688 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,689 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:42,689 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:42,689 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,689 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:42,689 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:42,690 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:42,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,690 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:42,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:42,693 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:42,693 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:42,693 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:42,693 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:42,693 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:42,696 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,696 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,696 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,697 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,698 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:42,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:42,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:42,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,702 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,702 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,703 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:42,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,703 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:42,703 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/cef04524ddf241f6952d09fc84072677 2023-04-16 20:55:42,803 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:42,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:42,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:42,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:42,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:42,809 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:42,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:42,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:42,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,809 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,810 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,811 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,811 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,811 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:42,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:42,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:42,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:42,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:42,816 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,816 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:42,816 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:42,816 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,818 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:42,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:42,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc30e33[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:42,820 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:42,820 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,823 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:42,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,824 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:42,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:42,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,829 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:42,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:42,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:42,833 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:42,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:42,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:42,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:42,833 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:42,835 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,835 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,835 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,842 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:42,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:42,847 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:42,847 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:42,947 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:42,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:42,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:42,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:42,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:42,951 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:42,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:42,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:42,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,953 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,954 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:42,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:42,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:42,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:42,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:42,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:42,955 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:42,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:42,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:42,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:42,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:42,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:42,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:42,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:42,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:42,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:42,960 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,960 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:42,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:42,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6f67b8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:42,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:42,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:42,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:42,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:42,963 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,963 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:42,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,966 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:42,966 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,966 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:42,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:42,970 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:42,970 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:42,970 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:42,970 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:42,970 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:42,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:42,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678602966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678602968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678602968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,973 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,974 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:42,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:42,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:42,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:42,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:42,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:42,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:42,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:42,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:42,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:42,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:42,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:42,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:42,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:42,979 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:42,979 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:42,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:42,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36102 deadline: 1681678602984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678603054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,079 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:43,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:43,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:43,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:43,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:43,084 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:43,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:43,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:43,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,086 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,086 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:43,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:43,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:43,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:43,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:43,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:43,088 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,088 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:43,088 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:43,088 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:43,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:43,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:43,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:43,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:43,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:43,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ccc242c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:43,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:43,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,098 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:43,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:43,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:43,099 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,099 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:43,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:43,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,100 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:43,100 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:43,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:43,102 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:43,103 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:43,103 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:43,103 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:43,103 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:43,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,109 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,111 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,111 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,111 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,111 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:43,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:43,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:43,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,111 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,112 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,112 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:43,112 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:43,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/d4fd18a9cb9b49afb51fdd617f9e34d7 2023-04-16 20:55:43,212 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:43,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:43,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:43,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:43,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:43,216 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:43,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:43,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:43,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,218 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,218 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,219 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:43,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:43,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:43,234 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:43,234 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,234 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:43,234 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:43,234 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:43,235 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,235 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:43,235 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:43,235 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:43,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:43,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:43,236 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:43,237 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,237 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,237 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,238 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/0bc86aba28c14087b3e25e3e6f294320 2023-04-16 20:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:43,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119f73eb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:43,242 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:43,242 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,245 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:43,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:43,247 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/cef04524ddf241f6952d09fc84072677 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cef04524ddf241f6952d09fc84072677 2023-04-16 20:55:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:43,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:43,252 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:43,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:43,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:43,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:43,252 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:43,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cef04524ddf241f6952d09fc84072677, entries=200, sequenceid=283, filesize=14.3 K 2023-04-16 20:55:43,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/d4fd18a9cb9b49afb51fdd617f9e34d7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d4fd18a9cb9b49afb51fdd617f9e34d7 2023-04-16 20:55:43,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d4fd18a9cb9b49afb51fdd617f9e34d7, entries=150, sequenceid=283, filesize=11.9 K 2023-04-16 20:55:43,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/0bc86aba28c14087b3e25e3e6f294320 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0bc86aba28c14087b3e25e3e6f294320 2023-04-16 20:55:43,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:43,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:43,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,263 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:43,264 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:43,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0bc86aba28c14087b3e25e3e6f294320, entries=150, sequenceid=283, filesize=11.9 K 2023-04-16 20:55:43,271 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 41b019418b53f9031c0d7e1ee3de5f99 in 953ms, sequenceid=283, compaction requested=false 2023-04-16 20:55:43,271 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:43,364 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:43,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:43,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:43,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:43,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:43,369 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:43,369 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,370 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:43,370 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:43,370 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,370 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,371 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,372 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,372 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:43,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:43,374 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:43,377 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,377 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:43,378 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-16 20:55:43,378 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:43,378 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:43,378 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:43,378 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:43,378 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:43,378 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:43,410 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/d717e91aa7e6494994275983623651a2 2023-04-16 20:55:43,435 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/281f2bdebf4c4095ba28e15d8793251f 2023-04-16 20:55:43,452 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/55947e04c28c4788b27f10ef359269b9 2023-04-16 20:55:43,459 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/d717e91aa7e6494994275983623651a2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d717e91aa7e6494994275983623651a2 2023-04-16 20:55:43,467 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d717e91aa7e6494994275983623651a2, entries=150, sequenceid=301, filesize=11.9 K 2023-04-16 20:55:43,469 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/281f2bdebf4c4095ba28e15d8793251f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/281f2bdebf4c4095ba28e15d8793251f 2023-04-16 20:55:43,472 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/281f2bdebf4c4095ba28e15d8793251f, entries=150, sequenceid=301, filesize=11.9 K 2023-04-16 20:55:43,473 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/55947e04c28c4788b27f10ef359269b9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/55947e04c28c4788b27f10ef359269b9 2023-04-16 20:55:43,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:43,478 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:43,478 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/55947e04c28c4788b27f10ef359269b9, entries=150, sequenceid=301, filesize=11.9 K 2023-04-16 20:55:43,478 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-16 20:55:43,479 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=6.71 KB/6870 for 41b019418b53f9031c0d7e1ee3de5f99 in 101ms, sequenceid=301, compaction requested=true 2023-04-16 20:55:43,479 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:43,479 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:43,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:43,479 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,480 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:43,480 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:43,482 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,483 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,483 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:43,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7448557d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:43,484 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:43,484 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,485 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,486 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:43,486 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:43,486 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:43,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:43,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:43,489 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:43,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:43,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:43,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:43,489 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:43,491 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,491 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,491 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,491 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:43,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:43,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:43,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:43,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:43,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:43,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:43,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,497 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,498 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,499 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,499 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,499 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,499 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,499 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,499 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,499 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:43,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:43,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:43,499 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:43,499 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:43,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/d57ec4e941694f55b5b6d1df9f378769 2023-04-16 20:55:43,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/12297f921ed9447ca4019bda256b61a1 2023-04-16 20:55:43,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/9cc6125742854beea50da3626bf60361 2023-04-16 20:55:43,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/d57ec4e941694f55b5b6d1df9f378769 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d57ec4e941694f55b5b6d1df9f378769 2023-04-16 20:55:43,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d57ec4e941694f55b5b6d1df9f378769, entries=200, sequenceid=312, filesize=14.3 K 2023-04-16 20:55:43,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/12297f921ed9447ca4019bda256b61a1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/12297f921ed9447ca4019bda256b61a1 2023-04-16 20:55:43,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/12297f921ed9447ca4019bda256b61a1, entries=150, sequenceid=312, filesize=11.9 K 2023-04-16 20:55:43,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/9cc6125742854beea50da3626bf60361 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cc6125742854beea50da3626bf60361 2023-04-16 20:55:43,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cc6125742854beea50da3626bf60361, entries=150, sequenceid=312, filesize=11.9 K 2023-04-16 20:55:43,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=114.05 KB/116790 for 41b019418b53f9031c0d7e1ee3de5f99 in 51ms, sequenceid=312, compaction requested=true 2023-04-16 20:55:43,544 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:43,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:43,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:43,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:55:43,545 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:43,544 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:43,546 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:43,546 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53898 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:43,546 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:43,546 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,546 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/196c679e36d74459b65e6ca8e5399a82, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cef04524ddf241f6952d09fc84072677, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d717e91aa7e6494994275983623651a2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d57ec4e941694f55b5b6d1df9f378769] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=52.6 K 2023-04-16 20:55:43,546 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:43,546 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,547 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/01c65d6e4fea4a34b0a0d3cd1824cf50, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d4fd18a9cb9b49afb51fdd617f9e34d7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/281f2bdebf4c4095ba28e15d8793251f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/12297f921ed9447ca4019bda256b61a1] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=47.9 K 2023-04-16 20:55:43,547 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 196c679e36d74459b65e6ca8e5399a82, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678541186 2023-04-16 20:55:43,547 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 01c65d6e4fea4a34b0a0d3cd1824cf50, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678541186 2023-04-16 20:55:43,547 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting cef04524ddf241f6952d09fc84072677, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681678541901 2023-04-16 20:55:43,547 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d4fd18a9cb9b49afb51fdd617f9e34d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681678541901 2023-04-16 20:55:43,548 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 281f2bdebf4c4095ba28e15d8793251f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681678542343 2023-04-16 20:55:43,548 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d717e91aa7e6494994275983623651a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681678542343 2023-04-16 20:55:43,548 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 12297f921ed9447ca4019bda256b61a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681678543487 2023-04-16 20:55:43,548 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d57ec4e941694f55b5b6d1df9f378769, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681678543482 2023-04-16 20:55:43,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678543528, max timestamp=1681678543546Num uniques 859; 2023-04-16 20:55:43,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:43,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678543528, max timestamp=1681678543546Num uniques 859; 2023-04-16 20:55:43,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:43,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678543528, max timestamp=1681678543546Num uniques 859; 2023-04-16 20:55:43,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:43,557 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#479 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:43,562 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#480 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:43,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-16 20:55:43,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:43,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:43,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:43,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:43,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:43,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:43,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:43,599 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:43,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/be4c148e03424e47bd19813fb0698549 2023-04-16 20:55:43,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:43,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:43,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:43,604 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/63844bb989674af38e17f84e362f3b4b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/63844bb989674af38e17f84e362f3b4b 2023-04-16 20:55:43,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:43,604 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:43,604 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,604 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:43,604 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:43,605 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,605 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,606 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/102de6e4a4ab4345b4f8f4bacface028 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/102de6e4a4ab4345b4f8f4bacface028 2023-04-16 20:55:43,606 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,606 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:43,612 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into 63844bb989674af38e17f84e362f3b4b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:43,612 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:43,612 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=12, startTime=1681678543544; duration=0sec 2023-04-16 20:55:43,612 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:43,612 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:43,613 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 102de6e4a4ab4345b4f8f4bacface028(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:43,613 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:43,613 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=12, startTime=1681678543544; duration=0sec 2023-04-16 20:55:43,613 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:43,614 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:43,614 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:43,614 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,614 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e118f4ec2f0e458eb016f161d2024508, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0bc86aba28c14087b3e25e3e6f294320, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/55947e04c28c4788b27f10ef359269b9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cc6125742854beea50da3626bf60361] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=47.9 K 2023-04-16 20:55:43,615 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e118f4ec2f0e458eb016f161d2024508, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678541186 2023-04-16 20:55:43,615 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0bc86aba28c14087b3e25e3e6f294320, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681678541901 2023-04-16 20:55:43,615 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 55947e04c28c4788b27f10ef359269b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681678542343 2023-04-16 20:55:43,616 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9cc6125742854beea50da3626bf60361, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681678543487 2023-04-16 20:55:43,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:43,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:43,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678543597, max timestamp=1681678543615Num uniques 859; 2023-04-16 20:55:43,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:43,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678543597, max timestamp=1681678543615Num uniques 859; 2023-04-16 20:55:43,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:43,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678543597, max timestamp=1681678543615Num uniques 859; 2023-04-16 20:55:43,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:43,620 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,620 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:43,620 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:43,620 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:43,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,624 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,624 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/8115389151b54303a9ceb00ea5e34ff1 2023-04-16 20:55:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,628 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,628 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:43,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658baeeb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:43,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:43,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,634 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#483 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:43,634 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,634 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:43,634 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:43,634 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:43,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,635 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:43,635 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:43,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,636 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,640 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/65fbae91bcaa467f870179c64ab75e5a 2023-04-16 20:55:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:43,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:43,642 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:43,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:43,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:43,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:43,642 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:43,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/be4c148e03424e47bd19813fb0698549 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4c148e03424e47bd19813fb0698549 2023-04-16 20:55:43,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4c148e03424e47bd19813fb0698549, entries=200, sequenceid=337, filesize=14.3 K 2023-04-16 20:55:43,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/8115389151b54303a9ceb00ea5e34ff1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/8115389151b54303a9ceb00ea5e34ff1 2023-04-16 20:55:43,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,653 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,656 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,659 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,659 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,659 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,659 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,659 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,659 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,659 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:43,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:43,659 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,659 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,660 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:43,660 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:43,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,659 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:43,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,664 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/32060b74cacd4773aae281ac9941373e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/32060b74cacd4773aae281ac9941373e 2023-04-16 20:55:43,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/8115389151b54303a9ceb00ea5e34ff1, entries=150, sequenceid=337, filesize=11.9 K 2023-04-16 20:55:43,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/65fbae91bcaa467f870179c64ab75e5a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/65fbae91bcaa467f870179c64ab75e5a 2023-04-16 20:55:43,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,671 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into 32060b74cacd4773aae281ac9941373e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:43,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678603655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,671 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:43,671 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=12, startTime=1681678543544; duration=0sec 2023-04-16 20:55:43,672 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:43,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/65fbae91bcaa467f870179c64ab75e5a, entries=150, sequenceid=337, filesize=11.9 K 2023-04-16 20:55:43,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=129.71 KB/132820 for 41b019418b53f9031c0d7e1ee3de5f99 in 108ms, sequenceid=337, compaction requested=false 2023-04-16 20:55:43,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:43,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:43,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=136.42 KB heapSize=390.06 KB 2023-04-16 20:55:43,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:43,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:43,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:43,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:43,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:43,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:43,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678603691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678603693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678603698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,760 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:43,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:43,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:43,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:43,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:43,763 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:43,764 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,764 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:43,764 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:43,764 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,764 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,765 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,766 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:43,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:43,767 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:43,767 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,767 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:43,776 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,776 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:43,776 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:43,776 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:43,780 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,780 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678603772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,781 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,782 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:43,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:43,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@585b1b32[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:43,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:43,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:43,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:43,785 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:43,785 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,785 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:43,786 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:43,786 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,786 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,786 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:43,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:43,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:43,790 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:43,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:43,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:43,791 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:43,791 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:43,792 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,792 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,792 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,793 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,794 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,800 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,800 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,800 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:43,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:43,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,801 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:43,801 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:43,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678603799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678603801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36120 deadline: 1681678603809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,901 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:43,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:43,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:43,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:43,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:43,905 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:43,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:43,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:43,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,905 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,907 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:43,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:43,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:43,908 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:43,908 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:43,908 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:43,909 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:43,909 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:43,910 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:43,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:43,911 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:43,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:43,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:43,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:43,911 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:43,913 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,913 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,913 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,913 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:43,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32b12bf9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:43,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:43,915 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:43,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:43,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:43,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,917 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:43,920 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:43,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:43,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:43,922 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:43,922 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:43,922 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:43,922 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:43,922 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:43,924 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,924 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,924 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:43,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:43,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:43,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:43,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:43,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:43,929 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:43,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:43,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:43,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:43,930 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:43,930 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:43,930 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:43,930 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:43,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:43,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:43,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:43,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678603982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678604008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678604008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678604015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,030 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:44,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:44,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:44,034 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:44,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:44,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:44,035 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,035 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,036 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,036 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,036 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,036 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:44,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:44,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:44,038 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,038 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:44,038 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:44,038 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:44,041 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,041 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,042 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,042 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:44,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@464736f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:44,043 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:44,043 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,046 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,046 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:44,046 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:44,046 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:44,046 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,046 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:44,048 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:44,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:44,051 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:44,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:44,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:44,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:44,051 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:44,053 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,053 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,053 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,054 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,055 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,061 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,061 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:44,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:44,062 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:44,062 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:44,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678604069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,088 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/a6aef94a541f445ca2e1972ba94afcce 2023-04-16 20:55:44,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/c7b8d621252d4bbea603f58b0775a419 2023-04-16 20:55:44,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/49e1d92161d4436abb251db319cd2183 2023-04-16 20:55:44,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/a6aef94a541f445ca2e1972ba94afcce as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/a6aef94a541f445ca2e1972ba94afcce 2023-04-16 20:55:44,123 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/a6aef94a541f445ca2e1972ba94afcce, entries=200, sequenceid=377, filesize=14.3 K 2023-04-16 20:55:44,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/c7b8d621252d4bbea603f58b0775a419 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7b8d621252d4bbea603f58b0775a419 2023-04-16 20:55:44,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7b8d621252d4bbea603f58b0775a419, entries=150, sequenceid=377, filesize=11.9 K 2023-04-16 20:55:44,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/49e1d92161d4436abb251db319cd2183 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/49e1d92161d4436abb251db319cd2183 2023-04-16 20:55:44,131 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/49e1d92161d4436abb251db319cd2183, entries=150, sequenceid=377, filesize=11.9 K 2023-04-16 20:55:44,132 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~390.02 KB/399376, currentSize=46.96 KB/48090 for 41b019418b53f9031c0d7e1ee3de5f99 in 453ms, sequenceid=377, compaction requested=true 2023-04-16 20:55:44,132 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:44,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:44,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:44,132 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:44,132 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:44,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:44,133 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41848 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:44,133 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:44,133 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:44,133 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:44,133 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,133 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,133 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/63844bb989674af38e17f84e362f3b4b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4c148e03424e47bd19813fb0698549, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/a6aef94a541f445ca2e1972ba94afcce] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=40.9 K 2023-04-16 20:55:44,133 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/102de6e4a4ab4345b4f8f4bacface028, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/8115389151b54303a9ceb00ea5e34ff1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7b8d621252d4bbea603f58b0775a419] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=36.1 K 2023-04-16 20:55:44,134 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 63844bb989674af38e17f84e362f3b4b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681678543487 2023-04-16 20:55:44,134 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 102de6e4a4ab4345b4f8f4bacface028, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681678543487 2023-04-16 20:55:44,135 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting be4c148e03424e47bd19813fb0698549, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681678543533 2023-04-16 20:55:44,135 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8115389151b54303a9ceb00ea5e34ff1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681678543533 2023-04-16 20:55:44,135 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting a6aef94a541f445ca2e1972ba94afcce, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681678543666 2023-04-16 20:55:44,136 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c7b8d621252d4bbea603f58b0775a419, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681678543666 2023-04-16 20:55:44,142 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:44,153 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#489 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:44,162 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:44,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:44,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:44,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:44,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:44,197 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:44,197 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,197 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:44,197 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:44,198 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,198 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,201 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,201 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,202 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,202 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:44,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,204 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/530d9262c5ad48fa84d1fad482f4de3e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/530d9262c5ad48fa84d1fad482f4de3e 2023-04-16 20:55:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:44,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:44,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:44,208 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:44,208 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,208 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:44,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:44,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:44,210 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,210 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:44,210 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:55:44,210 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:44,210 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:44,210 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:44,210 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:44,210 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:44,210 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:44,213 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into 530d9262c5ad48fa84d1fad482f4de3e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:44,213 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:44,213 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=13, startTime=1681678544132; duration=0sec 2023-04-16 20:55:44,213 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:44,213 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:44,215 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:44,216 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:44,216 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,216 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/32060b74cacd4773aae281ac9941373e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/65fbae91bcaa467f870179c64ab75e5a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/49e1d92161d4436abb251db319cd2183] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=36.1 K 2023-04-16 20:55:44,216 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 32060b74cacd4773aae281ac9941373e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681678543487 2023-04-16 20:55:44,217 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 65fbae91bcaa467f870179c64ab75e5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681678543533 2023-04-16 20:55:44,217 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 49e1d92161d4436abb251db319cd2183, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681678543666 2023-04-16 20:55:44,233 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:44,289 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:44,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:44,296 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/1062c1403cc24b5ba93208590c1e8907 2023-04-16 20:55:44,314 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/02b729cd9fd94943992f20b2c8128fb4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/02b729cd9fd94943992f20b2c8128fb4 2023-04-16 20:55:44,318 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into 02b729cd9fd94943992f20b2c8128fb4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:44,318 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:44,318 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=13, startTime=1681678544132; duration=0sec 2023-04-16 20:55:44,318 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:44,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678544327, max timestamp=1681678544345Num uniques 859; 2023-04-16 20:55:44,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:55:44,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678544327, max timestamp=1681678544345Num uniques 859; 2023-04-16 20:55:44,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:55:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678544336, max timestamp=1681678544347Num uniques 859; 2023-04-16 20:55:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:44,354 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/c7ee64ecffdf453b9b7b81a5dd280347 2023-04-16 20:55:44,381 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/67082986436b4d7794ed6d6d8bbb6b3f 2023-04-16 20:55:44,386 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/1062c1403cc24b5ba93208590c1e8907 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1062c1403cc24b5ba93208590c1e8907 2023-04-16 20:55:44,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678604382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,389 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1062c1403cc24b5ba93208590c1e8907, entries=150, sequenceid=387, filesize=11.9 K 2023-04-16 20:55:44,390 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/c7ee64ecffdf453b9b7b81a5dd280347 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7ee64ecffdf453b9b7b81a5dd280347 2023-04-16 20:55:44,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678604383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678604383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678604383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,394 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7ee64ecffdf453b9b7b81a5dd280347, entries=150, sequenceid=387, filesize=11.9 K 2023-04-16 20:55:44,395 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/67082986436b4d7794ed6d6d8bbb6b3f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/67082986436b4d7794ed6d6d8bbb6b3f 2023-04-16 20:55:44,401 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/67082986436b4d7794ed6d6d8bbb6b3f, entries=150, sequenceid=387, filesize=11.9 K 2023-04-16 20:55:44,401 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=145.36 KB/148850 for 41b019418b53f9031c0d7e1ee3de5f99 in 191ms, sequenceid=387, compaction requested=false 2023-04-16 20:55:44,401 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:44,401 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,402 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:44,402 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:44,402 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,402 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:44,402 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:44,404 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,404 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,405 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:44,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:44,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c2079fe[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:44,406 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:44,406 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:44,409 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,411 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:44,415 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:44,415 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:44,415 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:44,415 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:44,415 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:44,417 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,417 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,417 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,418 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,419 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,421 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,421 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,421 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,421 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,421 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:44,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:44,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:44,422 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,422 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:44,422 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:44,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:44,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=152.07 KB heapSize=431 KB 2023-04-16 20:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:55:44,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/26105fc4a465449fb4f4bccbc7d4fb45 2023-04-16 20:55:44,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678604503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678604503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678604505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678604513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,522 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:44,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:44,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:44,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:44,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:44,532 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:44,532 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,532 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:44,532 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:44,533 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,533 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,535 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,536 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,536 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:44,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:44,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:44,540 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,540 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:44,540 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:44,540 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:44,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:44,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:44,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:44,542 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,542 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca267c3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:44,546 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:44,546 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,547 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,548 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,548 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:44,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:44,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:44,548 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:44,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:44,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,550 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:44,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:44,557 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:44,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:44,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:44,557 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:44,557 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:44,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,560 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,562 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,566 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:44,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:44,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,566 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,566 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:44,566 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:44,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,586 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/5771bf15b2d34c85acc2b2690d01c4e4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5771bf15b2d34c85acc2b2690d01c4e4 2023-04-16 20:55:44,591 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 5771bf15b2d34c85acc2b2690d01c4e4(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:44,591 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:44,591 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=13, startTime=1681678544132; duration=0sec 2023-04-16 20:55:44,591 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:44,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678604615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678604615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678604616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678604621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,666 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:44,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:44,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:44,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:44,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:44,670 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:44,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:44,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:44,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,671 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,671 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:44,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:44,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:44,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:44,673 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,673 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:44,673 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:44,673 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:44,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:44,675 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f77d4d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:44,676 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:44,676 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,680 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,680 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:44,680 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:44,680 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:44,680 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,680 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:44,681 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:44,681 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,681 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,681 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:44,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:44,684 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:44,684 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:44,684 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:44,684 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:44,684 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:44,686 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,686 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,686 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,687 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,688 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,694 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,694 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:44,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,694 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:44,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,700 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:44,700 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:44,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,800 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:44,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:44,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:44,804 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:44,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:44,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:44,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,806 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,806 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,806 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,806 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:44,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:44,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:44,816 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,816 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:44,817 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:44,817 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:44,817 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:44,818 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,819 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:44,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202f89d7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:44,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:44,822 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,825 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:44,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:44,825 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:44,827 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,827 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,828 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:44,832 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:44,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:44,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:44,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:44,832 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:44,834 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,834 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,834 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,834 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678604826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678604826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678604827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678604829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,837 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,839 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:44,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:44,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,840 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:44,840 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:44,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/37da9158e8e74710a7c9f7e3ddab57a0 2023-04-16 20:55:44,940 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:44,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/ff0628dea4374fa596e8542b31388efd 2023-04-16 20:55:44,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:44,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:44,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:44,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:44,943 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:44,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:44,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:44,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,943 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,946 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:44,947 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:44,947 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,948 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:44,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:44,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:44,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:44,948 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:44,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:44,949 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,949 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:44,949 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:44,949 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:44,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:44,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:44,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:44,950 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,950 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:44,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/26105fc4a465449fb4f4bccbc7d4fb45 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/26105fc4a465449fb4f4bccbc7d4fb45 2023-04-16 20:55:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/26105fc4a465449fb4f4bccbc7d4fb45, entries=200, sequenceid=427, filesize=14.3 K 2023-04-16 20:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/37da9158e8e74710a7c9f7e3ddab57a0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/37da9158e8e74710a7c9f7e3ddab57a0 2023-04-16 20:55:44,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:44,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f7feba5[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:44,958 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:44,958 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,959 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,959 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:44,959 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:44,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:44,959 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:44,961 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/37da9158e8e74710a7c9f7e3ddab57a0, entries=150, sequenceid=427, filesize=11.9 K 2023-04-16 20:55:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/ff0628dea4374fa596e8542b31388efd as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff0628dea4374fa596e8542b31388efd 2023-04-16 20:55:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:44,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:44,966 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:44,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:44,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:44,967 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:44,967 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:44,969 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff0628dea4374fa596e8542b31388efd, entries=150, sequenceid=427, filesize=11.9 K 2023-04-16 20:55:44,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:44,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:44,969 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,969 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~152.07 KB/155720, heapSize ~430.95 KB/441296, currentSize=33.54 KB/34350 for 41b019418b53f9031c0d7e1ee3de5f99 in 474ms, sequenceid=427, compaction requested=true 2023-04-16 20:55:44,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,970 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:44,970 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:44,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:44,970 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:44,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:44,970 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:44,971 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:44,971 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:44,971 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:44,971 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:44,971 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,971 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,971 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5771bf15b2d34c85acc2b2690d01c4e4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7ee64ecffdf453b9b7b81a5dd280347, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/37da9158e8e74710a7c9f7e3ddab57a0] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=36.2 K 2023-04-16 20:55:44,971 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:44,972 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/530d9262c5ad48fa84d1fad482f4de3e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1062c1403cc24b5ba93208590c1e8907, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/26105fc4a465449fb4f4bccbc7d4fb45] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=38.6 K 2023-04-16 20:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,972 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5771bf15b2d34c85acc2b2690d01c4e4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681678543666 2023-04-16 20:55:44,972 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 530d9262c5ad48fa84d1fad482f4de3e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681678543666 2023-04-16 20:55:44,972 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c7ee64ecffdf453b9b7b81a5dd280347, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681678543689 2023-04-16 20:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:44,972 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1062c1403cc24b5ba93208590c1e8907, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681678543689 2023-04-16 20:55:44,973 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 37da9158e8e74710a7c9f7e3ddab57a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681678544372 2023-04-16 20:55:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:44,973 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 26105fc4a465449fb4f4bccbc7d4fb45, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681678544372 2023-04-16 20:55:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,974 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,974 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:44,974 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:44,974 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:44,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:44,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:44,974 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:44,974 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:44,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:44,974 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:44,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:44,974 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:44,974 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:44,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:44,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:44,978 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:44,980 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:45,031 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/7b386962e855422ebe66e10e6352a702 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/7b386962e855422ebe66e10e6352a702 2023-04-16 20:55:45,036 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 7b386962e855422ebe66e10e6352a702(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:45,036 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:45,036 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=13, startTime=1681678544970; duration=0sec 2023-04-16 20:55:45,036 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:45,036 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:45,037 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:45,037 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:45,037 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:45,037 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/02b729cd9fd94943992f20b2c8128fb4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/67082986436b4d7794ed6d6d8bbb6b3f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff0628dea4374fa596e8542b31388efd] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=36.2 K 2023-04-16 20:55:45,037 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 02b729cd9fd94943992f20b2c8128fb4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681678543666 2023-04-16 20:55:45,038 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 67082986436b4d7794ed6d6d8bbb6b3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681678543689 2023-04-16 20:55:45,038 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ff0628dea4374fa596e8542b31388efd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681678544372 2023-04-16 20:55:45,044 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#499 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:45,075 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:45,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:45,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:45,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:45,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:45,079 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:45,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:45,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:45,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:45,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:45,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:45,081 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:45,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:45,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:45,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:45,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:45,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:45,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:45,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:45,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:45,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:45,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:45,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:45,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:45,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:45,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:45,084 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-16 20:55:45,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:45,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:45,085 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:45,085 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:45,085 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:45,085 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:45,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:45,142 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:45,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678545161, max timestamp=1681678545171Num uniques 859; 2023-04-16 20:55:45,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678545161, max timestamp=1681678545171Num uniques 859; 2023-04-16 20:55:45,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:45,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:45,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678605197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678605197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678605205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678605206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678605307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678605307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678605316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678605316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,423 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/bba763e714a4487dbde642aa0df9943b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/bba763e714a4487dbde642aa0df9943b 2023-04-16 20:55:45,428 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into bba763e714a4487dbde642aa0df9943b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:45,428 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:45,428 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=13, startTime=1681678544970; duration=0sec 2023-04-16 20:55:45,428 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:45,455 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/e9c8d4003bdc4c56a5ff782451c24dad as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e9c8d4003bdc4c56a5ff782451c24dad 2023-04-16 20:55:45,460 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into e9c8d4003bdc4c56a5ff782451c24dad(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:45,460 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:45,460 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=13, startTime=1681678544970; duration=0sec 2023-04-16 20:55:45,460 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:45,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678605513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678605514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,518 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/342372cb50f642959a4a6a543c8edbce 2023-04-16 20:55:45,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678605522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678605523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,534 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/9829d5818eba4ad2944dc17d64c9e3f6 2023-04-16 20:55:45,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678605819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678605821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:45,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678605830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678605830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,952 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/2daa7bcd54c74d31a9cf3fea3c9760b6 2023-04-16 20:55:45,955 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/342372cb50f642959a4a6a543c8edbce as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/342372cb50f642959a4a6a543c8edbce 2023-04-16 20:55:45,960 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/342372cb50f642959a4a6a543c8edbce, entries=150, sequenceid=437, filesize=11.9 K 2023-04-16 20:55:45,961 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/9829d5818eba4ad2944dc17d64c9e3f6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9829d5818eba4ad2944dc17d64c9e3f6 2023-04-16 20:55:45,964 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9829d5818eba4ad2944dc17d64c9e3f6, entries=150, sequenceid=437, filesize=11.9 K 2023-04-16 20:55:45,965 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/2daa7bcd54c74d31a9cf3fea3c9760b6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2daa7bcd54c74d31a9cf3fea3c9760b6 2023-04-16 20:55:45,971 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2daa7bcd54c74d31a9cf3fea3c9760b6, entries=150, sequenceid=437, filesize=11.9 K 2023-04-16 20:55:45,972 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=143.13 KB/146560 for 41b019418b53f9031c0d7e1ee3de5f99 in 888ms, sequenceid=437, compaction requested=false 2023-04-16 20:55:45,972 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:45,972 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:45,973 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:45,973 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,973 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:45,973 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:45,974 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:45,975 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:45,975 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:45,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:45,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@304c1bf6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:45,976 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:45,976 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:45,978 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:45,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:45,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:45,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:45,978 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,978 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:45,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:45,978 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:45,980 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:45,982 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:45,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:45,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:45,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:45,982 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:45,983 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:45,983 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:45,983 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:45,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:45,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:45,984 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:45,985 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:45,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:45,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,987 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,987 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:45,987 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:45,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:45,987 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:45,987 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:45,987 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:45,987 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:45,987 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:45,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:45,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:45,987 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:45,987 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:45,988 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:45,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:45,988 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:45,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:46,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:46,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=149.83 KB heapSize=433.31 KB 2023-04-16 20:55:46,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:46,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:46,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:46,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:46,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:46,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:46,088 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:46,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:46,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:46,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:46,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:46,092 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:46,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:46,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:46,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,095 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:46,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:46,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:46,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:46,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:46,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:46,097 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,097 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:46,097 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:46,097 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,097 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:46,097 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:46,101 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,102 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:46,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ced15a7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:46,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:46,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,106 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:46,107 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,107 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:46,108 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:46,108 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:46,108 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,108 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:46,112 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:46,112 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:46,112 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:46,112 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:46,112 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:46,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:46,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678606110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,120 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,120 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:46,120 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,124 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,127 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,129 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:46,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:46,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:46,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:46,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:46,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:46,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:46,130 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:46,130 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:46,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:46,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:46,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678606218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,230 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:46,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:46,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:46,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:46,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:46,234 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:46,234 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,234 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:46,234 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:46,234 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,234 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,235 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:46,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:46,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,236 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:46,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:46,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:46,238 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,238 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:46,238 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:46,238 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,238 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:46,238 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:46,238 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,238 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:46,238 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:46,239 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:46,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50d054dc[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:46,242 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:46,242 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,244 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:46,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:46,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:46,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:46,245 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:46,245 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:46,245 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:46,245 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:46,248 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:46,248 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:46,248 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:46,248 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:46,248 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:46,250 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,250 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:46,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,250 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,253 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:46,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:46,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:46,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:46,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:46,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:46,257 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:46,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,257 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:46,257 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:46,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:46,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678606328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:46,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678606328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:46,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678606338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:46,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678606341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,357 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:46,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:46,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:46,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:46,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:46,360 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:46,360 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:46,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:46,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,363 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:46,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:46,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:46,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:46,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:46,372 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,372 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:46,372 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:46,372 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:46,374 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3092afa0[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:46,375 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:46,375 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,376 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:46,377 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,377 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:46,378 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:46,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:46,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:46,381 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:46,381 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:46,381 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:46,381 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:46,381 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:46,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,383 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:46,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,384 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,385 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:46,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:46,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:46,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:46,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:46,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:46,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:46,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:46,390 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,391 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,391 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,391 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:46,391 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:46,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:46,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:46,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678606426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,491 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:46,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:46,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:46,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:46,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:46,494 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:46,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:46,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:46,495 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,495 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:46,496 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,497 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,497 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:46,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:46,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:46,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:46,498 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,498 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:46,498 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:46,498 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:46,501 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,501 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,501 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,501 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:46,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@605181ef[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:46,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:46,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/752fd36a4bce42cb988f09bf9a2fe078 2023-04-16 20:55:46,507 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:46,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:46,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,509 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:46,509 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:46,509 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:46,515 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:46,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:46,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:46,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:46,515 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:46,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/fbd51e9a68f84b7086f50199c770b6a6 2023-04-16 20:55:46,517 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,517 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,517 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,518 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:46,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=69.33 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/1ae80adb6b7444f39fc1f83ed3e0aaa2 2023-04-16 20:55:46,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:46,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:46,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:46,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:46,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:46,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:46,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:46,527 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:46,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,528 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:46,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:46,528 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:46,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/752fd36a4bce42cb988f09bf9a2fe078 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/752fd36a4bce42cb988f09bf9a2fe078 2023-04-16 20:55:46,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/752fd36a4bce42cb988f09bf9a2fe078, entries=200, sequenceid=473, filesize=14.3 K 2023-04-16 20:55:46,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/fbd51e9a68f84b7086f50199c770b6a6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fbd51e9a68f84b7086f50199c770b6a6 2023-04-16 20:55:46,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fbd51e9a68f84b7086f50199c770b6a6, entries=150, sequenceid=473, filesize=11.9 K 2023-04-16 20:55:46,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/1ae80adb6b7444f39fc1f83ed3e0aaa2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1ae80adb6b7444f39fc1f83ed3e0aaa2 2023-04-16 20:55:46,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1ae80adb6b7444f39fc1f83ed3e0aaa2, entries=150, sequenceid=473, filesize=11.9 K 2023-04-16 20:55:46,545 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~433.27 KB/443664, currentSize=26.84 KB/27480 for 41b019418b53f9031c0d7e1ee3de5f99 in 460ms, sequenceid=473, compaction requested=true 2023-04-16 20:55:46,545 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:46,546 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:46,546 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:46,546 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:46,546 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:46,546 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:46,547 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:46,547 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:46,547 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,547 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/bba763e714a4487dbde642aa0df9943b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/342372cb50f642959a4a6a543c8edbce, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/752fd36a4bce42cb988f09bf9a2fe078] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=38.7 K 2023-04-16 20:55:46,547 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:46,547 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:46,547 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,547 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/7b386962e855422ebe66e10e6352a702, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9829d5818eba4ad2944dc17d64c9e3f6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fbd51e9a68f84b7086f50199c770b6a6] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=36.3 K 2023-04-16 20:55:46,547 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting bba763e714a4487dbde642aa0df9943b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681678544372 2023-04-16 20:55:46,548 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7b386962e855422ebe66e10e6352a702, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681678544372 2023-04-16 20:55:46,548 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 342372cb50f642959a4a6a543c8edbce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681678544499 2023-04-16 20:55:46,548 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9829d5818eba4ad2944dc17d64c9e3f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681678544499 2023-04-16 20:55:46,548 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 752fd36a4bce42cb988f09bf9a2fe078, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1681678545192 2023-04-16 20:55:46,549 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting fbd51e9a68f84b7086f50199c770b6a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1681678545196 2023-04-16 20:55:46,558 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:46,561 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#507 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:46,576 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/5a961a332e8b482da430bf83486bb68d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a961a332e8b482da430bf83486bb68d 2023-04-16 20:55:46,580 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 5a961a332e8b482da430bf83486bb68d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:46,580 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:46,580 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=13, startTime=1681678546546; duration=0sec 2023-04-16 20:55:46,580 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:46,580 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:46,581 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:46,581 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:46,581 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,581 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e9c8d4003bdc4c56a5ff782451c24dad, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2daa7bcd54c74d31a9cf3fea3c9760b6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1ae80adb6b7444f39fc1f83ed3e0aaa2] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=36.3 K 2023-04-16 20:55:46,581 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e9c8d4003bdc4c56a5ff782451c24dad, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681678544372 2023-04-16 20:55:46,582 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 2daa7bcd54c74d31a9cf3fea3c9760b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681678544499 2023-04-16 20:55:46,582 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1ae80adb6b7444f39fc1f83ed3e0aaa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1681678545196 2023-04-16 20:55:46,587 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:46,604 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/85676f3aefd645fcb22771687f503b3d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/85676f3aefd645fcb22771687f503b3d 2023-04-16 20:55:46,610 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into 85676f3aefd645fcb22771687f503b3d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:46,610 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:46,610 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=13, startTime=1681678546546; duration=0sec 2023-04-16 20:55:46,610 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:46,628 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:46,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:46,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:46,631 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:46,632 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,632 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:46,632 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:46,632 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,632 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,633 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:46,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:46,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:46,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:46,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:46,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:46,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:46,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:46,636 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:46,636 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 20:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:46,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:46,740 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:46,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678546768, max timestamp=1681678546800Num uniques 859; 2023-04-16 20:55:46,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678546768, max timestamp=1681678546800Num uniques 859; 2023-04-16 20:55:46,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:46,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678546768, max timestamp=1681678546800Num uniques 859; 2023-04-16 20:55:46,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:46,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:46,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678546848, max timestamp=1681678546874Num uniques 1018; 2023-04-16 20:55:46,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 20:55:46,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678546848, max timestamp=1681678546874Num uniques 1018; 2023-04-16 20:55:46,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 20:55:46,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681678546848, max timestamp=1681678546874Num uniques 1018; 2023-04-16 20:55:46,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 20:55:46,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681678546928, max timestamp=1681678546970Num uniques 1027; 2023-04-16 20:55:46,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681678546928, max timestamp=1681678546970Num uniques 1027; 2023-04-16 20:55:46,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-16 20:55:46,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-16 20:55:46,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681678546928, max timestamp=1681678546970Num uniques 1027; 2023-04-16 20:55:46,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-16 20:55:46,986 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/d553c74e97824bed8718d141e55bfa95 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d553c74e97824bed8718d141e55bfa95 2023-04-16 20:55:46,990 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into d553c74e97824bed8718d141e55bfa95(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:46,990 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:46,990 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=13, startTime=1681678546546; duration=0sec 2023-04-16 20:55:46,990 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:47,046 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/be4592537c444f13ae7a657dfadd81d7 2023-04-16 20:55:47,055 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/cfa82e0f7980431eaaa017bc7fa3d6a2 2023-04-16 20:55:47,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678547026, max timestamp=1681678547059Num uniques 1036; 2023-04-16 20:55:47,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678547026, max timestamp=1681678547059Num uniques 1036; 2023-04-16 20:55:47,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-16 20:55:47,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678547026, max timestamp=1681678547059Num uniques 1036; 2023-04-16 20:55:47,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-16 20:55:47,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-16 20:55:47,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681678547126, max timestamp=1681678547150Num uniques 1045; 2023-04-16 20:55:47,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681678547126, max timestamp=1681678547150Num uniques 1045; 2023-04-16 20:55:47,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-16 20:55:47,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-16 20:55:47,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681678547126, max timestamp=1681678547150Num uniques 1045; 2023-04-16 20:55:47,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-16 20:55:47,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681678547214, max timestamp=1681678547235Num uniques 1104; 2023-04-16 20:55:47,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681678547214, max timestamp=1681678547235Num uniques 1104; 2023-04-16 20:55:47,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681678547214, max timestamp=1681678547235Num uniques 1104; 2023-04-16 20:55:47,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-16 20:55:47,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-16 20:55:47,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-16 20:55:47,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681678547299, max timestamp=1681678547337Num uniques 1013; 2023-04-16 20:55:47,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-16 20:55:47,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681678547299, max timestamp=1681678547337Num uniques 1013; 2023-04-16 20:55:47,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-16 20:55:47,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681678547299, max timestamp=1681678547337Num uniques 1013; 2023-04-16 20:55:47,340 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-16 20:55:47,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681678547359, max timestamp=1681678547375Num uniques 1022; 2023-04-16 20:55:47,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681678547359, max timestamp=1681678547375Num uniques 1022; 2023-04-16 20:55:47,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-16 20:55:47,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-16 20:55:47,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681678547359, max timestamp=1681678547375Num uniques 1022; 2023-04-16 20:55:47,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-16 20:55:47,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678607395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678607397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681678547392, max timestamp=1681678547401Num uniques 1131; 2023-04-16 20:55:47,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-04-16 20:55:47,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681678547392, max timestamp=1681678547401Num uniques 1131; 2023-04-16 20:55:47,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681678547392, max timestamp=1681678547401Num uniques 1131; 2023-04-16 20:55:47,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-04-16 20:55:47,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-04-16 20:55:47,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678607434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681678547430, max timestamp=1681678547438Num uniques 1140; 2023-04-16 20:55:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681678547430, max timestamp=1681678547438Num uniques 1140; 2023-04-16 20:55:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681678547430, max timestamp=1681678547438Num uniques 1140; 2023-04-16 20:55:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-16 20:55:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-16 20:55:47,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-16 20:55:47,466 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/578cdf66b13c4a658041586090f5d953 2023-04-16 20:55:47,470 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/be4592537c444f13ae7a657dfadd81d7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4592537c444f13ae7a657dfadd81d7 2023-04-16 20:55:47,473 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4592537c444f13ae7a657dfadd81d7, entries=150, sequenceid=482, filesize=11.9 K 2023-04-16 20:55:47,474 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/cfa82e0f7980431eaaa017bc7fa3d6a2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cfa82e0f7980431eaaa017bc7fa3d6a2 2023-04-16 20:55:47,478 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cfa82e0f7980431eaaa017bc7fa3d6a2, entries=150, sequenceid=482, filesize=11.9 K 2023-04-16 20:55:47,479 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/578cdf66b13c4a658041586090f5d953 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/578cdf66b13c4a658041586090f5d953 2023-04-16 20:55:47,481 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/578cdf66b13c4a658041586090f5d953, entries=150, sequenceid=482, filesize=11.9 K 2023-04-16 20:55:47,482 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=107.34 KB/109920 for 41b019418b53f9031c0d7e1ee3de5f99 in 846ms, sequenceid=482, compaction requested=false 2023-04-16 20:55:47,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:47,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:47,483 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,484 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,484 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:47,485 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=114.05 KB heapSize=317.63 KB 2023-04-16 20:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:47,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:47,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:47,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:47,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:47,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f5f06f[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:47,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:47,488 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,489 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:47,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,489 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:47,491 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:47,494 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:47,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:47,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:47,494 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:47,494 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:47,495 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,496 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,497 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,497 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:47,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/31ee7548ee9b427590cc91b6baba890f 2023-04-16 20:55:47,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,499 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,502 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,502 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,502 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,502 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,502 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,502 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,502 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:47,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:47,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:47,502 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:47,502 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:47,503 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:47,503 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:47,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:47,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:47,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678607510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678607514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678607514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678607518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678607541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,602 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:47,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:47,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:47,605 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:47,605 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:47,605 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:47,605 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:47,605 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,605 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,607 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:47,608 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:47,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:47,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:47,610 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:47,610 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:47,610 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:47,610 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:47,613 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,613 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,613 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,613 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d13299[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:47,614 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:47,615 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,616 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,616 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,616 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:47,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:47,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:47,622 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:47,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:47,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:47,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:47,622 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:47,623 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,624 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,624 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:47,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,625 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:47,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,628 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:47,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:47,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:47,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:47,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:47,628 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,629 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:47,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:47,629 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:47,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:47,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678607620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678607620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678607621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678607625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,729 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:47,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:47,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:47,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:47,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:47,732 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:47,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:47,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:47,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:47,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,734 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:47,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:47,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:47,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:47,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:47,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:47,735 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:47,735 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:47,735 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:47,735 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:47,737 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,738 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,738 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,738 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731b784c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:47,739 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:47,739 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,740 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:47,740 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,740 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:47,744 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:47,744 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:47,744 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,744 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:47,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:47,747 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:47,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:47,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:47,747 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:47,747 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:47,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,748 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:47,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:47,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678607746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,752 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,757 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,757 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:47,757 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:47,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:47,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:47,757 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:47,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:47,758 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:47,758 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:47,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:47,757 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,758 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,758 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,758 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,758 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:47,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:47,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678607832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678607832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678607832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:47,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678607832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,858 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:47,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:47,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:47,861 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:47,861 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:47,861 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:47,861 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:47,861 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,862 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,863 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,863 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:47,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:47,863 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,863 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:47,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:47,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:47,864 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:47,865 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:47,865 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:47,865 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:47,865 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:47,869 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,869 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,869 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,869 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f29d139[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:47,870 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:47,871 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,872 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:47,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:47,873 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:47,875 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:47,875 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:47,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:47,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:47,876 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:47,877 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,877 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:47,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,877 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:47,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:47,879 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,882 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,882 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:47,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:47,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:47,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:47,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:47,883 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:47,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:47,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,883 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:47,887 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:47,887 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:47,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/519231234abc4289967dc14adb9869e6 2023-04-16 20:55:47,987 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:47,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:47,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:47,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:47,991 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,992 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:47,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:47,993 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,994 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,994 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:47,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:47,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:47,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:47,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:47,995 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:47,995 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:47,995 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:47,995 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,995 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:47,997 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,997 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:48,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:48,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34f42c3b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:48,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:48,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,002 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:48,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:48,005 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:48,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:48,005 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:48,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:48,006 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:48,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:48,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:48,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:48,006 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:48,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:48,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:48,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:48,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:48,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:48,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:48,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:48,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:48,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:48,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,012 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:48,012 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:48,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:48,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:48,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678608050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,112 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:48,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:48,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:48,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:48,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:48,115 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:48,116 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:48,116 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:48,116 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:48,116 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,116 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,117 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,118 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:48,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:48,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,118 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:48,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:48,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:48,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:48,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:48,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:48,120 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:48,121 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:48,121 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:48,121 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:48,121 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:48,121 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:48,121 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,121 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:48,121 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:48,124 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b6e85c6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:48,129 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:48,129 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,130 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:48,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:48,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:48,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:48,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:48,135 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:48,135 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:48,135 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:48,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:48,136 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:48,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:48,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,137 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:48,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:48,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:48,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678608137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678608137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678608138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678608139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,146 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,146 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:48,146 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:48,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:48,146 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:48,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:48,146 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:48,147 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:48,147 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:48,146 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:48,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,247 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:48,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:48,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:48,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:48,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:48,250 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:48,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:48,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:48,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:48,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,252 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:48,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:48,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:48,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:48,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:48,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:48,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:48,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:48,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:48,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:48,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:48,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:48,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@431f49ff[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:48,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:48,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,259 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:48,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:48,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:48,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:48,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:48,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:48,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:48,265 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:48,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:48,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:48,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:48,265 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:48,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:48,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:48,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:48,267 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,269 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:48,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:48,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:48,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:48,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:48,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:48,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:48,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,272 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:48,272 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:48,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:48,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:48,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/6b197222845d4b80bae5c7aa2e529007 2023-04-16 20:55:48,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/31ee7548ee9b427590cc91b6baba890f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/31ee7548ee9b427590cc91b6baba890f 2023-04-16 20:55:48,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/31ee7548ee9b427590cc91b6baba890f, entries=200, sequenceid=670, filesize=14.3 K 2023-04-16 20:55:48,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/519231234abc4289967dc14adb9869e6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/519231234abc4289967dc14adb9869e6 2023-04-16 20:55:48,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/519231234abc4289967dc14adb9869e6, entries=150, sequenceid=670, filesize=11.9 K 2023-04-16 20:55:48,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/6b197222845d4b80bae5c7aa2e529007 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b197222845d4b80bae5c7aa2e529007 2023-04-16 20:55:48,347 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b197222845d4b80bae5c7aa2e529007, entries=150, sequenceid=670, filesize=11.9 K 2023-04-16 20:55:48,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~317.58 KB/325200, currentSize=67.09 KB/68700 for 41b019418b53f9031c0d7e1ee3de5f99 in 862ms, sequenceid=670, compaction requested=true 2023-04-16 20:55:48,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:48,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:48,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:55:48,348 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-16 20:55:48,348 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:48,348 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:48,349 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:48,349 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:48,349 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:48,349 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/85676f3aefd645fcb22771687f503b3d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/578cdf66b13c4a658041586090f5d953, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b197222845d4b80bae5c7aa2e529007] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=36.4 K 2023-04-16 20:55:48,349 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:48,349 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:48,349 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 85676f3aefd645fcb22771687f503b3d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1681678545196 2023-04-16 20:55:48,349 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:48,349 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d553c74e97824bed8718d141e55bfa95, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4592537c444f13ae7a657dfadd81d7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/31ee7548ee9b427590cc91b6baba890f] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=38.8 K 2023-04-16 20:55:48,349 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 578cdf66b13c4a658041586090f5d953, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681678546096 2023-04-16 20:55:48,350 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d553c74e97824bed8718d141e55bfa95, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1681678545196 2023-04-16 20:55:48,350 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6b197222845d4b80bae5c7aa2e529007, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681678547466 2023-04-16 20:55:48,351 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting be4592537c444f13ae7a657dfadd81d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681678546096 2023-04-16 20:55:48,351 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 31ee7548ee9b427590cc91b6baba890f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681678547466 2023-04-16 20:55:48,364 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#515 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:48,365 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#516 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:48,372 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:48,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:48,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:48,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:48,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:48,384 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,384 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,386 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:48,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:48,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:48,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:48,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:48,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:48,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:48,400 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:48,400 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:48,400 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:48,408 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:48,409 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:48,409 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:55:48,409 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:48,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:48,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:48,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:48,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:48,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:48,413 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/6b3a0ac95c9c414d9e9ee06673b628fe as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b3a0ac95c9c414d9e9ee06673b628fe 2023-04-16 20:55:48,418 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into 6b3a0ac95c9c414d9e9ee06673b628fe(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:48,418 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:48,418 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=13, startTime=1681678548348; duration=0sec 2023-04-16 20:55:48,418 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:48,419 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:48,419 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:48,420 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:48,420 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:48,420 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a961a332e8b482da430bf83486bb68d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cfa82e0f7980431eaaa017bc7fa3d6a2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/519231234abc4289967dc14adb9869e6] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=36.4 K 2023-04-16 20:55:48,420 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5a961a332e8b482da430bf83486bb68d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1681678545196 2023-04-16 20:55:48,421 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting cfa82e0f7980431eaaa017bc7fa3d6a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681678546096 2023-04-16 20:55:48,421 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 519231234abc4289967dc14adb9869e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681678547466 2023-04-16 20:55:48,427 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/1ec96878fd684992b09c7d525fa05daf as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1ec96878fd684992b09c7d525fa05daf 2023-04-16 20:55:48,429 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:48,444 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into 1ec96878fd684992b09c7d525fa05daf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:48,444 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:48,444 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=13, startTime=1681678548347; duration=0sec 2023-04-16 20:55:48,444 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:48,453 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/61b58583d9a748c7abe91945b9e5ffd5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/61b58583d9a748c7abe91945b9e5ffd5 2023-04-16 20:55:48,459 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 61b58583d9a748c7abe91945b9e5ffd5(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:48,459 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:48,459 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=13, startTime=1681678548347; duration=0sec 2023-04-16 20:55:48,459 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:48,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:48,564 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:48,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678608634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678608643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678608644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678608646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678608648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678608740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:48,845 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/afa967c907b14f4497d06b5ca90c32d5 2023-04-16 20:55:48,857 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/e956eff68f5248949c3895ae8f437c10 2023-04-16 20:55:48,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:48,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678608943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678609248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,269 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/74eadf50db8444fa837094ddd435bc0c 2023-04-16 20:55:49,273 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/afa967c907b14f4497d06b5ca90c32d5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/afa967c907b14f4497d06b5ca90c32d5 2023-04-16 20:55:49,276 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/afa967c907b14f4497d06b5ca90c32d5, entries=150, sequenceid=683, filesize=11.9 K 2023-04-16 20:55:49,277 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/e956eff68f5248949c3895ae8f437c10 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e956eff68f5248949c3895ae8f437c10 2023-04-16 20:55:49,280 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e956eff68f5248949c3895ae8f437c10, entries=150, sequenceid=683, filesize=11.9 K 2023-04-16 20:55:49,281 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/74eadf50db8444fa837094ddd435bc0c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/74eadf50db8444fa837094ddd435bc0c 2023-04-16 20:55:49,284 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/74eadf50db8444fa837094ddd435bc0c, entries=150, sequenceid=683, filesize=11.9 K 2023-04-16 20:55:49,284 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 41b019418b53f9031c0d7e1ee3de5f99 in 875ms, sequenceid=683, compaction requested=false 2023-04-16 20:55:49,284 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:49,284 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:49,284 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:49,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:49,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:49,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:49,286 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:49,286 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:49,287 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,287 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12eee8bc[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:49,289 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:49,289 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,290 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:49,290 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,291 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:49,292 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:49,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:49,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:49,292 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:49,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:49,295 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:49,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:49,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:49,295 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:49,295 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:49,298 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,298 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,298 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:49,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:49,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:49,299 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,301 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,303 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:49,304 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:49,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:49,304 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:49,304 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:49,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:49,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:49,304 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,304 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:49,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:49,304 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,304 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:49,304 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,305 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,305 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,305 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,405 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:49,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:49,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:49,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:49,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:49,408 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:49,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:49,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:49,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:49,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,409 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:49,409 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:49,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:49,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:49,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:49,412 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:49,412 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:49,412 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-16 20:55:49,412 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:49,412 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:49,412 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:49,412 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:49,412 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:49,413 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:49,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:49,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:49,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678609663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678609665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678609666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678609667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678609757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678609768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678609770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678609776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:49,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678609776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,820 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/7a96802d18464f05873b1e7d63b2be8c 2023-04-16 20:55:49,831 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/9fd7de6c33574a6fbad5b9b7c1ceb484 2023-04-16 20:55:49,840 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/25c0398bb7b94b95a7abcfda517dd36e 2023-04-16 20:55:49,844 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/7a96802d18464f05873b1e7d63b2be8c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7a96802d18464f05873b1e7d63b2be8c 2023-04-16 20:55:49,847 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7a96802d18464f05873b1e7d63b2be8c, entries=150, sequenceid=705, filesize=11.9 K 2023-04-16 20:55:49,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/9fd7de6c33574a6fbad5b9b7c1ceb484 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9fd7de6c33574a6fbad5b9b7c1ceb484 2023-04-16 20:55:49,852 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9fd7de6c33574a6fbad5b9b7c1ceb484, entries=150, sequenceid=705, filesize=11.9 K 2023-04-16 20:55:49,852 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/25c0398bb7b94b95a7abcfda517dd36e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/25c0398bb7b94b95a7abcfda517dd36e 2023-04-16 20:55:49,857 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/25c0398bb7b94b95a7abcfda517dd36e, entries=150, sequenceid=705, filesize=11.9 K 2023-04-16 20:55:49,858 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 41b019418b53f9031c0d7e1ee3de5f99 in 446ms, sequenceid=705, compaction requested=true 2023-04-16 20:55:49,858 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:49,858 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:49,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:49,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:49,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:49,858 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:49,860 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:49,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,860 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:49,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:49,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e9b2e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:49,862 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:49,862 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,864 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:49,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:49,865 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:49,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,865 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:49,866 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,866 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:49,866 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:49,866 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:49,869 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:49,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:49,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:49,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:49,869 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:49,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,870 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:49,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:49,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:49,871 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:49,873 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,878 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,878 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:49,878 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:49,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:49,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:49,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:49,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:49,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,879 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:49,879 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:49,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:49,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:49,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:49,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 20:55:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:49,979 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:49,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:49,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:49,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:49,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:49,982 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:49,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:49,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:49,982 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:49,983 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,983 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,985 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:49,985 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:49,986 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:49,986 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:49,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:49,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/d6ad78d4773241d284e234254f2b1571 2023-04-16 20:55:49,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:49,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:49,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:49,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:49,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:49,987 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,989 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18317eb0[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:49,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:49,991 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,993 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:49,993 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,993 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:49,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/457c63fe6f44435b9fb9a5bcd7bb7da5 2023-04-16 20:55:49,999 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,999 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:49,999 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:49,999 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:50,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:50,005 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:50,005 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:50,005 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:50,005 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:50,005 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:50,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/365f0e43da3c460f9d0d5c90eefbf084 2023-04-16 20:55:50,007 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/d6ad78d4773241d284e234254f2b1571 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d6ad78d4773241d284e234254f2b1571 2023-04-16 20:55:50,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:50,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:50,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:50,011 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:50,011 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:50,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:50,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:50,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:50,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d6ad78d4773241d284e234254f2b1571, entries=200, sequenceid=719, filesize=14.3 K 2023-04-16 20:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/457c63fe6f44435b9fb9a5bcd7bb7da5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/457c63fe6f44435b9fb9a5bcd7bb7da5 2023-04-16 20:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678610003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:50,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/457c63fe6f44435b9fb9a5bcd7bb7da5, entries=150, sequenceid=719, filesize=11.9 K 2023-04-16 20:55:50,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/365f0e43da3c460f9d0d5c90eefbf084 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/365f0e43da3c460f9d0d5c90eefbf084 2023-04-16 20:55:50,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/365f0e43da3c460f9d0d5c90eefbf084, entries=150, sequenceid=719, filesize=11.9 K 2023-04-16 20:55:50,022 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 41b019418b53f9031c0d7e1ee3de5f99 in 45ms, sequenceid=719, compaction requested=true 2023-04-16 20:55:50,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:50,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:50,022 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:50,022 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:50,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:50,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:50,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:50,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-16 20:55:50,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:50,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:50,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:50,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:50,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:50,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:50,026 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:50,026 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:50,026 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,026 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/61b58583d9a748c7abe91945b9e5ffd5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e956eff68f5248949c3895ae8f437c10, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9fd7de6c33574a6fbad5b9b7c1ceb484, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/457c63fe6f44435b9fb9a5bcd7bb7da5] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=48.4 K 2023-04-16 20:55:50,027 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 61b58583d9a748c7abe91945b9e5ffd5, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681678547466 2023-04-16 20:55:50,027 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e956eff68f5248949c3895ae8f437c10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1681678547509 2023-04-16 20:55:50,027 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52002 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:50,027 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:50,027 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,027 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9fd7de6c33574a6fbad5b9b7c1ceb484, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1681678548621 2023-04-16 20:55:50,028 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1ec96878fd684992b09c7d525fa05daf, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/afa967c907b14f4497d06b5ca90c32d5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7a96802d18464f05873b1e7d63b2be8c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d6ad78d4773241d284e234254f2b1571] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=50.8 K 2023-04-16 20:55:50,028 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 457c63fe6f44435b9fb9a5bcd7bb7da5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681678549664 2023-04-16 20:55:50,029 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1ec96878fd684992b09c7d525fa05daf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681678547466 2023-04-16 20:55:50,029 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting afa967c907b14f4497d06b5ca90c32d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1681678547509 2023-04-16 20:55:50,029 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7a96802d18464f05873b1e7d63b2be8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1681678548621 2023-04-16 20:55:50,030 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d6ad78d4773241d284e234254f2b1571, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681678549653 2023-04-16 20:55:50,033 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=739 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/50e57fbf67a84d66b69267aee4c43a2f 2023-04-16 20:55:50,038 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#528 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:50,041 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:50,056 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/d28203e6a09149c4854d4c1db3ad356d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d28203e6a09149c4854d4c1db3ad356d 2023-04-16 20:55:50,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678610048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678610049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678610058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,062 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into d28203e6a09149c4854d4c1db3ad356d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:50,062 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:50,062 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=12, startTime=1681678550022; duration=0sec 2023-04-16 20:55:50,062 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:50,063 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/6ffc7930fe3743d2a8b78507fefd6a9c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6ffc7930fe3743d2a8b78507fefd6a9c 2023-04-16 20:55:50,063 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:50,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=739 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/a3ef677afcba4b529a12b3dfcaeae53a 2023-04-16 20:55:50,064 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:50,065 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:50,065 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,065 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b3a0ac95c9c414d9e9ee06673b628fe, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/74eadf50db8444fa837094ddd435bc0c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/25c0398bb7b94b95a7abcfda517dd36e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/365f0e43da3c460f9d0d5c90eefbf084] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=48.4 K 2023-04-16 20:55:50,066 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6b3a0ac95c9c414d9e9ee06673b628fe, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681678547466 2023-04-16 20:55:50,066 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 74eadf50db8444fa837094ddd435bc0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1681678547509 2023-04-16 20:55:50,067 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 25c0398bb7b94b95a7abcfda517dd36e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1681678548621 2023-04-16 20:55:50,067 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 365f0e43da3c460f9d0d5c90eefbf084, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681678549664 2023-04-16 20:55:50,068 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 6ffc7930fe3743d2a8b78507fefd6a9c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:50,068 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:50,068 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=12, startTime=1681678550022; duration=0sec 2023-04-16 20:55:50,068 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:50,085 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#532 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:50,102 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/1a861f79201548309e7edad1dc4e1a77 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1a861f79201548309e7edad1dc4e1a77 2023-04-16 20:55:50,106 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into 1a861f79201548309e7edad1dc4e1a77(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:50,106 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:50,107 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=12, startTime=1681678550023; duration=0sec 2023-04-16 20:55:50,107 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:50,111 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:50,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:50,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:50,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:50,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:50,114 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:50,114 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:50,114 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:50,115 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:50,115 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,115 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,116 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,116 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:50,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:50,117 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,117 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:50,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:50,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678610115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:50,125 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,125 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:50,125 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:50,125 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,125 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:50,125 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:50,125 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,125 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:50,125 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:50,127 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21588cf8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:50,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:50,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:50,132 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,132 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:50,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:50,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:50,133 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:50,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:50,137 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:50,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:50,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:50,137 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:50,137 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:50,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:50,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:50,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:50,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:50,144 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:50,144 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:50,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:50,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:50,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:50,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:50,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678610158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678610161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678610162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,244 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:50,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:50,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:50,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:50,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:50,247 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:50,247 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:50,247 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:50,247 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:50,248 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,248 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,249 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:50,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:50,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,249 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:50,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:50,250 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:50,250 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:50,250 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:50,251 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,251 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:50,251 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:50,251 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,253 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:50,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44163b67[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:50,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:50,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:50,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:50,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:50,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:50,259 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,259 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:50,261 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:50,261 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:50,261 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:50,261 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:50,261 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:50,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:50,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:50,264 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:50,268 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:50,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:50,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:50,269 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,269 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,269 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,269 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,270 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:50,270 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:50,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:50,270 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,270 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:50,270 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:50,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678610321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678610362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,370 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:50,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:50,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678610368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678610368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:50,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:50,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:50,374 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:50,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:50,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:50,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:50,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,380 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:50,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:50,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:50,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:50,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:50,381 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:50,381 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:50,381 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:50,382 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,382 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:50,382 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:50,382 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:50,385 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,385 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,385 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,385 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37526f20[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:50,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:50,387 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,387 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:50,388 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:50,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:50,388 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:50,389 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,389 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:50,389 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,389 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:50,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:50,392 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:50,392 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:50,392 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:50,392 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:50,392 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:50,395 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,395 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,395 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:50,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,396 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,397 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:50,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:50,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:50,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:50,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:50,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:50,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,399 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:50,399 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:50,400 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:50,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=739 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/4531fdb165dc4f1abbfc18f36c70e35d 2023-04-16 20:55:50,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/50e57fbf67a84d66b69267aee4c43a2f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/50e57fbf67a84d66b69267aee4c43a2f 2023-04-16 20:55:50,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/50e57fbf67a84d66b69267aee4c43a2f, entries=250, sequenceid=739, filesize=16.7 K 2023-04-16 20:55:50,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/a3ef677afcba4b529a12b3dfcaeae53a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a3ef677afcba4b529a12b3dfcaeae53a 2023-04-16 20:55:50,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a3ef677afcba4b529a12b3dfcaeae53a, entries=150, sequenceid=739, filesize=11.9 K 2023-04-16 20:55:50,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/4531fdb165dc4f1abbfc18f36c70e35d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4531fdb165dc4f1abbfc18f36c70e35d 2023-04-16 20:55:50,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4531fdb165dc4f1abbfc18f36c70e35d, entries=150, sequenceid=739, filesize=11.9 K 2023-04-16 20:55:50,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 41b019418b53f9031c0d7e1ee3de5f99 in 471ms, sequenceid=739, compaction requested=false 2023-04-16 20:55:50,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:50,500 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:50,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:50,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:50,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:50,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:50,503 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:50,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:50,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:50,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:50,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,503 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,504 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,504 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:50,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:50,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:50,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:50,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:50,506 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,506 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:50,506 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 20:55:50,506 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:50,507 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:50,507 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:50,507 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:50,507 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:50,507 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:50,543 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/58ec70ec19484784914f05457d2cce53 2023-04-16 20:55:50,564 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/fb5eca62a8884159a6077f463294b9d4 2023-04-16 20:55:50,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:50,628 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:50,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678610678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678610679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678610680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678610681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36120 deadline: 1681678610763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36054 deadline: 1681678610783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681678610788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678610788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:50,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678610788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,975 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/b35efe4e9d304f3081e2542c5715d369 2023-04-16 20:55:50,978 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/58ec70ec19484784914f05457d2cce53 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/58ec70ec19484784914f05457d2cce53 2023-04-16 20:55:50,982 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/58ec70ec19484784914f05457d2cce53, entries=150, sequenceid=754, filesize=11.9 K 2023-04-16 20:55:50,982 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/fb5eca62a8884159a6077f463294b9d4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fb5eca62a8884159a6077f463294b9d4 2023-04-16 20:55:50,985 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fb5eca62a8884159a6077f463294b9d4, entries=150, sequenceid=754, filesize=11.9 K 2023-04-16 20:55:50,986 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/b35efe4e9d304f3081e2542c5715d369 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/b35efe4e9d304f3081e2542c5715d369 2023-04-16 20:55:50,989 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/b35efe4e9d304f3081e2542c5715d369, entries=150, sequenceid=754, filesize=11.9 K 2023-04-16 20:55:50,989 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 41b019418b53f9031c0d7e1ee3de5f99 in 483ms, sequenceid=754, compaction requested=true 2023-04-16 20:55:50,990 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:50,990 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:50,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:50,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:50,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:50,990 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:50,991 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,991 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:50,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:50,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24854384[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:50,994 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:50,994 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,995 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:50,996 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:50,996 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:50,996 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:50,996 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,996 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:50,997 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678550678, max timestamp=1681678550999Num uniques 859; 2023-04-16 20:55:51,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:51,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:51,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:51,001 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-04-16 20:55:51,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678550678, max timestamp=1681678550999Num uniques 859; 2023-04-16 20:55:51,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678550678, max timestamp=1681678550999Num uniques 859; 2023-04-16 20:55:51,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:51,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:55:51,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:51,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:51,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:51,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:51,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:51,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:51,004 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:51,004 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:51,004 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:51,004 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:51,004 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:51,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:51,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:51,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:51,008 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,015 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:51,015 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:51,015 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,015 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:51,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:51,016 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,016 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:51,015 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:51,016 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:51,016 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:51,016 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:51,016 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:51,016 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678551019, max timestamp=1681678551037Num uniques 859; 2023-04-16 20:55:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678551019, max timestamp=1681678551037Num uniques 859; 2023-04-16 20:55:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681678551019, max timestamp=1681678551037Num uniques 859; 2023-04-16 20:55:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:55:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:55:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 20:55:51,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:51,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36096 deadline: 1681678611075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681678551065, max timestamp=1681678551078Num uniques 1118; 2023-04-16 20:55:51,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681678551065, max timestamp=1681678551078Num uniques 1118; 2023-04-16 20:55:51,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681678551065, max timestamp=1681678551078Num uniques 1118; 2023-04-16 20:55:51,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-16 20:55:51,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-16 20:55:51,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-16 20:55:51,116 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:51,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:51,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:51,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:51,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:51,122 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:51,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:51,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:51,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:51,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,127 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:51,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:51,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,128 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:51,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:51,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:51,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:51,129 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:51,129 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:51,129 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:51,130 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:51,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:51,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:51,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:51,130 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:51,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,131 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:51,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:51,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41df24a0[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:51,133 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:51,133 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681678551106, max timestamp=1681678551133Num uniques 1077; 2023-04-16 20:55:51,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-16 20:55:51,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681678551106, max timestamp=1681678551133Num uniques 1077; 2023-04-16 20:55:51,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-16 20:55:51,136 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:51,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:51,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:51,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681678551106, max timestamp=1681678551133Num uniques 1077; 2023-04-16 20:55:51,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-16 20:55:51,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:51,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:51,138 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:51,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:51,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:51,138 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:51,139 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:51,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:51,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:51,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:51,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:51,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:51,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:51,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:51,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,147 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:51,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:51,147 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:51,147 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:51,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:51,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:51,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,148 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,148 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:51,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36102 deadline: 1681678611180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678551163, max timestamp=1681678551186Num uniques 1136; 2023-04-16 20:55:51,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678551163, max timestamp=1681678551186Num uniques 1136; 2023-04-16 20:55:51,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-16 20:55:51,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-16 20:55:51,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681678551163, max timestamp=1681678551186Num uniques 1136; 2023-04-16 20:55:51,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-16 20:55:51,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681678551218, max timestamp=1681678551235Num uniques 1045; 2023-04-16 20:55:51,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-16 20:55:51,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681678551218, max timestamp=1681678551235Num uniques 1045; 2023-04-16 20:55:51,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-16 20:55:51,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681678551218, max timestamp=1681678551235Num uniques 1045; 2023-04-16 20:55:51,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-16 20:55:51,247 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:51,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:51,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:51,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:51,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:51,250 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:51,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:51,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:51,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:51,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,252 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:51,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:51,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:51,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:51,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:51,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:51,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:51,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:51,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:51,253 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:51,253 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. started... 2023-04-16 20:55:51,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. as already flushing 2023-04-16 20:55:51,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:51,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:51,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:51,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:51,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e7f2ff[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:51,257 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:51,257 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:51,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:51,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:51,263 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:51,263 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:51,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:51,266 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:51,266 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:51,266 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:51,266 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:51,266 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:51,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:51,267 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:51,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:51,268 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,269 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:51,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:51,273 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:51,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:51,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:51,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:51,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:51,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:51,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:51,273 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:51,273 DEBUG [Thread-3179] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:51,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:51,273 DEBUG [Thread-3179] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:51,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:51,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681678551269, max timestamp=1681678551292Num uniques 1054; 2023-04-16 20:55:51,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-16 20:55:51,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681678551269, max timestamp=1681678551292Num uniques 1054; 2023-04-16 20:55:51,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-16 20:55:51,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681678551269, max timestamp=1681678551292Num uniques 1054; 2023-04-16 20:55:51,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-16 20:55:51,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7455, totalHeapSize=17192, min timestamp=1681678551313, max timestamp=1681678551345Num uniques 1113; 2023-04-16 20:55:51,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681678551313, max timestamp=1681678551345Num uniques 1113; 2023-04-16 20:55:51,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50965, new segment data size=7455, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-16 20:55:51,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-16 20:55:51,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7455, totalHeapSize=17192, min timestamp=1681678551313, max timestamp=1681678551345Num uniques 1113; 2023-04-16 20:55:51,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50965, new segment data size=7455, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-16 20:55:51,373 DEBUG [Thread-3188] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10a344e6 to 127.0.0.1:54578 2023-04-16 20:55:51,373 DEBUG [Thread-3188] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:51,373 DEBUG [Thread-3179] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:51,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:51,378 DEBUG [Thread-3177] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:54578 2023-04-16 20:55:51,379 DEBUG [Thread-3182] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:54578 2023-04-16 20:55:51,379 DEBUG [Thread-3177] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:51,379 DEBUG [Thread-3182] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:51,379 DEBUG [Thread-3186] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:54578 2023-04-16 20:55:51,379 DEBUG [Thread-3186] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:51,379 DEBUG [Thread-3173] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:54578 2023-04-16 20:55:51,379 DEBUG [Thread-3173] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:51,380 DEBUG [Thread-3175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:54578 2023-04-16 20:55:51,380 DEBUG [Thread-3175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:51,381 DEBUG [Thread-3171] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:54578 2023-04-16 20:55:51,381 DEBUG [Thread-3171] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:51,384 DEBUG [Thread-3184] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:54578 2023-04-16 20:55:51,384 DEBUG [Thread-3184] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:51,388 DEBUG [Thread-3180] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:54578 2023-04-16 20:55:51,388 DEBUG [Thread-3180] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:51,412 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/7d4395bfee4c473e93698de223305195 2023-04-16 20:55:51,820 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/5e6652a82ecc409b916a27fd5e0e1d9d 2023-04-16 20:55:52,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/fd52954ae91d49b5b93d9f29eec90444 2023-04-16 20:55:52,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/7d4395bfee4c473e93698de223305195 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7d4395bfee4c473e93698de223305195 2023-04-16 20:55:52,234 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7d4395bfee4c473e93698de223305195, entries=150, sequenceid=776, filesize=11.9 K 2023-04-16 20:55:52,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/5e6652a82ecc409b916a27fd5e0e1d9d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5e6652a82ecc409b916a27fd5e0e1d9d 2023-04-16 20:55:52,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5e6652a82ecc409b916a27fd5e0e1d9d, entries=150, sequenceid=776, filesize=11.9 K 2023-04-16 20:55:52,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/fd52954ae91d49b5b93d9f29eec90444 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/fd52954ae91d49b5b93d9f29eec90444 2023-04-16 20:55:52,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/fd52954ae91d49b5b93d9f29eec90444, entries=150, sequenceid=776, filesize=11.9 K 2023-04-16 20:55:52,241 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=107.34 KB/109920 for 41b019418b53f9031c0d7e1ee3de5f99 in 1240ms, sequenceid=776, compaction requested=true 2023-04-16 20:55:52,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:52,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:52,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:52,241 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:52,241 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:52,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:52,242 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54578 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:52,242 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:52,242 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/A is initiating minor compaction (all files) 2023-04-16 20:55:52,242 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/B is initiating minor compaction (all files) 2023-04-16 20:55:52,242 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/A in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:52,242 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/B in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:52,242 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d28203e6a09149c4854d4c1db3ad356d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/50e57fbf67a84d66b69267aee4c43a2f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/58ec70ec19484784914f05457d2cce53, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7d4395bfee4c473e93698de223305195] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=53.3 K 2023-04-16 20:55:52,242 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6ffc7930fe3743d2a8b78507fefd6a9c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a3ef677afcba4b529a12b3dfcaeae53a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fb5eca62a8884159a6077f463294b9d4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5e6652a82ecc409b916a27fd5e0e1d9d] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=48.5 K 2023-04-16 20:55:52,242 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d28203e6a09149c4854d4c1db3ad356d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681678549664 2023-04-16 20:55:52,242 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6ffc7930fe3743d2a8b78507fefd6a9c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681678549664 2023-04-16 20:55:52,243 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 50e57fbf67a84d66b69267aee4c43a2f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1681678550011 2023-04-16 20:55:52,243 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting a3ef677afcba4b529a12b3dfcaeae53a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1681678550011 2023-04-16 20:55:52,243 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 58ec70ec19484784914f05457d2cce53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1681678550045 2023-04-16 20:55:52,243 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting fb5eca62a8884159a6077f463294b9d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1681678550045 2023-04-16 20:55:52,243 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7d4395bfee4c473e93698de223305195, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1681678550680 2023-04-16 20:55:52,243 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5e6652a82ecc409b916a27fd5e0e1d9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1681678550680 2023-04-16 20:55:52,254 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#A#compaction#539 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:52,256 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#B#compaction#540 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:52,265 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/601775089b9d4bfba83ee9ffca58d3a8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/601775089b9d4bfba83ee9ffca58d3a8 2023-04-16 20:55:52,268 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/B of 41b019418b53f9031c0d7e1ee3de5f99 into 601775089b9d4bfba83ee9ffca58d3a8(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:52,268 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:52,268 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/B, priority=12, startTime=1681678552241; duration=0sec 2023-04-16 20:55:52,268 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:52,268 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:52,269 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:52,269 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 41b019418b53f9031c0d7e1ee3de5f99/C is initiating minor compaction (all files) 2023-04-16 20:55:52,269 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41b019418b53f9031c0d7e1ee3de5f99/C in TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:52,269 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1a861f79201548309e7edad1dc4e1a77, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4531fdb165dc4f1abbfc18f36c70e35d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/b35efe4e9d304f3081e2542c5715d369, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/fd52954ae91d49b5b93d9f29eec90444] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp, totalSize=48.5 K 2023-04-16 20:55:52,270 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1a861f79201548309e7edad1dc4e1a77, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681678549664 2023-04-16 20:55:52,270 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 4531fdb165dc4f1abbfc18f36c70e35d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1681678550011 2023-04-16 20:55:52,270 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b35efe4e9d304f3081e2542c5715d369, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1681678550045 2023-04-16 20:55:52,271 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting fd52954ae91d49b5b93d9f29eec90444, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1681678550680 2023-04-16 20:55:52,281 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 41b019418b53f9031c0d7e1ee3de5f99#C#compaction#541 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:52,665 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/d3772e489c4d4b5e91059c219f08edb8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d3772e489c4d4b5e91059c219f08edb8 2023-04-16 20:55:52,669 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/A of 41b019418b53f9031c0d7e1ee3de5f99 into d3772e489c4d4b5e91059c219f08edb8(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:52,669 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:52,669 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/A, priority=12, startTime=1681678552241; duration=0sec 2023-04-16 20:55:52,669 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:52,688 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/a37532671c03403490438bac04a104b2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/a37532671c03403490438bac04a104b2 2023-04-16 20:55:52,691 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 41b019418b53f9031c0d7e1ee3de5f99/C of 41b019418b53f9031c0d7e1ee3de5f99 into a37532671c03403490438bac04a104b2(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:52,691 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:52,691 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99., storeName=41b019418b53f9031c0d7e1ee3de5f99/C, priority=12, startTime=1681678552241; duration=0sec 2023-04-16 20:55:52,691 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:52,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:52,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41b019418b53f9031c0d7e1ee3de5f99 3/3 column families, dataSize=114.05 KB heapSize=324.84 KB 2023-04-16 20:55:52,773 DEBUG [Thread-3169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:54578 2023-04-16 20:55:52,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=A 2023-04-16 20:55:52,773 DEBUG [Thread-3169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:52,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:52,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=B 2023-04-16 20:55:52,773 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 20:55:52,773 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 140 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 144 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 232 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 165 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1769 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5307 rows 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1743 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5226 rows 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1749 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5247 rows 2023-04-16 20:55:52,774 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1744 2023-04-16 20:55:52,775 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5232 rows 2023-04-16 20:55:52,775 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(402): scanned 1764 2023-04-16 20:55:52,775 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(403): verified 5292 rows 2023-04-16 20:55:52,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:52,775 INFO [Listener at localhost/45179] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 20:55:52,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41b019418b53f9031c0d7e1ee3de5f99, store=C 2023-04-16 20:55:52,775 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec943a9 to 127.0.0.1:54578 2023-04-16 20:55:52,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:52,775 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:55:52,777 INFO [Listener at localhost/45179] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 20:55:52,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 20:55:52,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:52,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 20:55:52,783 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678552783"}]},"ts":"1681678552783"} 2023-04-16 20:55:52,784 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 20:55:52,786 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 20:55:52,787 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41b019418b53f9031c0d7e1ee3de5f99, UNASSIGN}] 2023-04-16 20:55:52,788 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41b019418b53f9031c0d7e1ee3de5f99, UNASSIGN 2023-04-16 20:55:52,788 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=41b019418b53f9031c0d7e1ee3de5f99, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:52,788 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678552788"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678552788"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678552788"}]},"ts":"1681678552788"} 2023-04-16 20:55:52,789 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:55:52,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 20:55:52,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:52,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 41b019418b53f9031c0d7e1ee3de5f99, disabling compactions & flushes 2023-04-16 20:55:52,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:53,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 20:55:53,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/61c0f8385c7146d484236e4b8787fb7c 2023-04-16 20:55:53,196 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/cd1d2bbfd25443ffadbf4a9a8f846bba 2023-04-16 20:55:53,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/ff06e8bf8574477296e88b9ef0179d53 2023-04-16 20:55:53,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/A/61c0f8385c7146d484236e4b8787fb7c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/61c0f8385c7146d484236e4b8787fb7c 2023-04-16 20:55:53,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/61c0f8385c7146d484236e4b8787fb7c, entries=150, sequenceid=916, filesize=11.9 K 2023-04-16 20:55:53,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/B/cd1d2bbfd25443ffadbf4a9a8f846bba as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cd1d2bbfd25443ffadbf4a9a8f846bba 2023-04-16 20:55:53,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cd1d2bbfd25443ffadbf4a9a8f846bba, entries=150, sequenceid=916, filesize=11.9 K 2023-04-16 20:55:53,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/.tmp/C/ff06e8bf8574477296e88b9ef0179d53 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff06e8bf8574477296e88b9ef0179d53 2023-04-16 20:55:53,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff06e8bf8574477296e88b9ef0179d53, entries=150, sequenceid=916, filesize=11.9 K 2023-04-16 20:55:53,225 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~324.80 KB/332592, currentSize=0 B/0 for 41b019418b53f9031c0d7e1ee3de5f99 in 452ms, sequenceid=916, compaction requested=false 2023-04-16 20:55:53,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:53,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:53,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:53,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. after waiting 0 ms 2023-04-16 20:55:53,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:53,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/51ccc339b80c4926ae30786a51fb2af0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e0bd6d5fca5e400bbe9ab530daa91f38, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e192d9baa24742f989acf118a793bd7a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/5c9d56f9d219470eb66bd6f4dbb6060d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/effe7881e062402789789e85f4e6731b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/f7ad5fbd640a4319b0984a2d108618f7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/9c1a3b69b0d94ced88a2c7afd4e0862f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1d0d9b47260d4046981e79a3f9f99fa5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/196c679e36d74459b65e6ca8e5399a82, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cef04524ddf241f6952d09fc84072677, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d717e91aa7e6494994275983623651a2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d57ec4e941694f55b5b6d1df9f378769, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/63844bb989674af38e17f84e362f3b4b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4c148e03424e47bd19813fb0698549, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/a6aef94a541f445ca2e1972ba94afcce, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/530d9262c5ad48fa84d1fad482f4de3e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1062c1403cc24b5ba93208590c1e8907, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/26105fc4a465449fb4f4bccbc7d4fb45, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/bba763e714a4487dbde642aa0df9943b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/342372cb50f642959a4a6a543c8edbce, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/752fd36a4bce42cb988f09bf9a2fe078, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d553c74e97824bed8718d141e55bfa95, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4592537c444f13ae7a657dfadd81d7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/31ee7548ee9b427590cc91b6baba890f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1ec96878fd684992b09c7d525fa05daf, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/afa967c907b14f4497d06b5ca90c32d5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7a96802d18464f05873b1e7d63b2be8c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d6ad78d4773241d284e234254f2b1571, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d28203e6a09149c4854d4c1db3ad356d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/50e57fbf67a84d66b69267aee4c43a2f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/58ec70ec19484784914f05457d2cce53, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7d4395bfee4c473e93698de223305195] to archive 2023-04-16 20:55:53,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:53,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/51ccc339b80c4926ae30786a51fb2af0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/51ccc339b80c4926ae30786a51fb2af0 2023-04-16 20:55:53,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e0bd6d5fca5e400bbe9ab530daa91f38 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e0bd6d5fca5e400bbe9ab530daa91f38 2023-04-16 20:55:53,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e192d9baa24742f989acf118a793bd7a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/e192d9baa24742f989acf118a793bd7a 2023-04-16 20:55:53,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/5c9d56f9d219470eb66bd6f4dbb6060d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/5c9d56f9d219470eb66bd6f4dbb6060d 2023-04-16 20:55:53,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/effe7881e062402789789e85f4e6731b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/effe7881e062402789789e85f4e6731b 2023-04-16 20:55:53,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/f7ad5fbd640a4319b0984a2d108618f7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/f7ad5fbd640a4319b0984a2d108618f7 2023-04-16 20:55:53,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/9c1a3b69b0d94ced88a2c7afd4e0862f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/9c1a3b69b0d94ced88a2c7afd4e0862f 2023-04-16 20:55:53,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1d0d9b47260d4046981e79a3f9f99fa5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1d0d9b47260d4046981e79a3f9f99fa5 2023-04-16 20:55:53,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/196c679e36d74459b65e6ca8e5399a82 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/196c679e36d74459b65e6ca8e5399a82 2023-04-16 20:55:53,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cef04524ddf241f6952d09fc84072677 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/cef04524ddf241f6952d09fc84072677 2023-04-16 20:55:53,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d717e91aa7e6494994275983623651a2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d717e91aa7e6494994275983623651a2 2023-04-16 20:55:53,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d57ec4e941694f55b5b6d1df9f378769 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d57ec4e941694f55b5b6d1df9f378769 2023-04-16 20:55:53,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/63844bb989674af38e17f84e362f3b4b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/63844bb989674af38e17f84e362f3b4b 2023-04-16 20:55:53,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4c148e03424e47bd19813fb0698549 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4c148e03424e47bd19813fb0698549 2023-04-16 20:55:53,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/a6aef94a541f445ca2e1972ba94afcce to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/a6aef94a541f445ca2e1972ba94afcce 2023-04-16 20:55:53,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/530d9262c5ad48fa84d1fad482f4de3e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/530d9262c5ad48fa84d1fad482f4de3e 2023-04-16 20:55:53,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1062c1403cc24b5ba93208590c1e8907 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1062c1403cc24b5ba93208590c1e8907 2023-04-16 20:55:53,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/26105fc4a465449fb4f4bccbc7d4fb45 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/26105fc4a465449fb4f4bccbc7d4fb45 2023-04-16 20:55:53,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/bba763e714a4487dbde642aa0df9943b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/bba763e714a4487dbde642aa0df9943b 2023-04-16 20:55:53,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/342372cb50f642959a4a6a543c8edbce to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/342372cb50f642959a4a6a543c8edbce 2023-04-16 20:55:53,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/752fd36a4bce42cb988f09bf9a2fe078 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/752fd36a4bce42cb988f09bf9a2fe078 2023-04-16 20:55:53,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d553c74e97824bed8718d141e55bfa95 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d553c74e97824bed8718d141e55bfa95 2023-04-16 20:55:53,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4592537c444f13ae7a657dfadd81d7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/be4592537c444f13ae7a657dfadd81d7 2023-04-16 20:55:53,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/31ee7548ee9b427590cc91b6baba890f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/31ee7548ee9b427590cc91b6baba890f 2023-04-16 20:55:53,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1ec96878fd684992b09c7d525fa05daf to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/1ec96878fd684992b09c7d525fa05daf 2023-04-16 20:55:53,255 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/afa967c907b14f4497d06b5ca90c32d5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/afa967c907b14f4497d06b5ca90c32d5 2023-04-16 20:55:53,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7a96802d18464f05873b1e7d63b2be8c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7a96802d18464f05873b1e7d63b2be8c 2023-04-16 20:55:53,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d6ad78d4773241d284e234254f2b1571 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d6ad78d4773241d284e234254f2b1571 2023-04-16 20:55:53,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d28203e6a09149c4854d4c1db3ad356d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d28203e6a09149c4854d4c1db3ad356d 2023-04-16 20:55:53,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/50e57fbf67a84d66b69267aee4c43a2f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/50e57fbf67a84d66b69267aee4c43a2f 2023-04-16 20:55:53,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/58ec70ec19484784914f05457d2cce53 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/58ec70ec19484784914f05457d2cce53 2023-04-16 20:55:53,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7d4395bfee4c473e93698de223305195 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/7d4395bfee4c473e93698de223305195 2023-04-16 20:55:53,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0729e563738e475a9fb461e7e46eb995, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c43a63cfefe6491f9d4214c406ad6b25, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/b613bbbe20c34034ae2189151b571211, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e7d9ca646ecf42b682482cc62445dc4a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/907499b4e85b461fa24d428d9bcb0259, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/84ea2eb67247476aad22d4db08df50c8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/02e0d2c2ed0e41eda3b62932614e5087, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/01c65d6e4fea4a34b0a0d3cd1824cf50, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/939caa7e695b40119520779da633dae6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d4fd18a9cb9b49afb51fdd617f9e34d7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/281f2bdebf4c4095ba28e15d8793251f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/102de6e4a4ab4345b4f8f4bacface028, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/12297f921ed9447ca4019bda256b61a1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/8115389151b54303a9ceb00ea5e34ff1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5771bf15b2d34c85acc2b2690d01c4e4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7b8d621252d4bbea603f58b0775a419, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7ee64ecffdf453b9b7b81a5dd280347, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/7b386962e855422ebe66e10e6352a702, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/37da9158e8e74710a7c9f7e3ddab57a0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9829d5818eba4ad2944dc17d64c9e3f6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a961a332e8b482da430bf83486bb68d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fbd51e9a68f84b7086f50199c770b6a6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cfa82e0f7980431eaaa017bc7fa3d6a2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/61b58583d9a748c7abe91945b9e5ffd5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/519231234abc4289967dc14adb9869e6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e956eff68f5248949c3895ae8f437c10, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9fd7de6c33574a6fbad5b9b7c1ceb484, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6ffc7930fe3743d2a8b78507fefd6a9c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/457c63fe6f44435b9fb9a5bcd7bb7da5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a3ef677afcba4b529a12b3dfcaeae53a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fb5eca62a8884159a6077f463294b9d4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5e6652a82ecc409b916a27fd5e0e1d9d] to archive 2023-04-16 20:55:53,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:53,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0729e563738e475a9fb461e7e46eb995 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/0729e563738e475a9fb461e7e46eb995 2023-04-16 20:55:53,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c43a63cfefe6491f9d4214c406ad6b25 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c43a63cfefe6491f9d4214c406ad6b25 2023-04-16 20:55:53,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/b613bbbe20c34034ae2189151b571211 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/b613bbbe20c34034ae2189151b571211 2023-04-16 20:55:53,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e7d9ca646ecf42b682482cc62445dc4a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e7d9ca646ecf42b682482cc62445dc4a 2023-04-16 20:55:53,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/907499b4e85b461fa24d428d9bcb0259 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/907499b4e85b461fa24d428d9bcb0259 2023-04-16 20:55:53,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/84ea2eb67247476aad22d4db08df50c8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/84ea2eb67247476aad22d4db08df50c8 2023-04-16 20:55:53,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/02e0d2c2ed0e41eda3b62932614e5087 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/02e0d2c2ed0e41eda3b62932614e5087 2023-04-16 20:55:53,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/01c65d6e4fea4a34b0a0d3cd1824cf50 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/01c65d6e4fea4a34b0a0d3cd1824cf50 2023-04-16 20:55:53,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/939caa7e695b40119520779da633dae6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/939caa7e695b40119520779da633dae6 2023-04-16 20:55:53,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d4fd18a9cb9b49afb51fdd617f9e34d7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/d4fd18a9cb9b49afb51fdd617f9e34d7 2023-04-16 20:55:53,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/281f2bdebf4c4095ba28e15d8793251f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/281f2bdebf4c4095ba28e15d8793251f 2023-04-16 20:55:53,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/102de6e4a4ab4345b4f8f4bacface028 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/102de6e4a4ab4345b4f8f4bacface028 2023-04-16 20:55:53,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/12297f921ed9447ca4019bda256b61a1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/12297f921ed9447ca4019bda256b61a1 2023-04-16 20:55:53,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/8115389151b54303a9ceb00ea5e34ff1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/8115389151b54303a9ceb00ea5e34ff1 2023-04-16 20:55:53,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5771bf15b2d34c85acc2b2690d01c4e4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5771bf15b2d34c85acc2b2690d01c4e4 2023-04-16 20:55:53,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7b8d621252d4bbea603f58b0775a419 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7b8d621252d4bbea603f58b0775a419 2023-04-16 20:55:53,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7ee64ecffdf453b9b7b81a5dd280347 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/c7ee64ecffdf453b9b7b81a5dd280347 2023-04-16 20:55:53,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/7b386962e855422ebe66e10e6352a702 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/7b386962e855422ebe66e10e6352a702 2023-04-16 20:55:53,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/37da9158e8e74710a7c9f7e3ddab57a0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/37da9158e8e74710a7c9f7e3ddab57a0 2023-04-16 20:55:53,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9829d5818eba4ad2944dc17d64c9e3f6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9829d5818eba4ad2944dc17d64c9e3f6 2023-04-16 20:55:53,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a961a332e8b482da430bf83486bb68d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5a961a332e8b482da430bf83486bb68d 2023-04-16 20:55:53,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fbd51e9a68f84b7086f50199c770b6a6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fbd51e9a68f84b7086f50199c770b6a6 2023-04-16 20:55:53,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cfa82e0f7980431eaaa017bc7fa3d6a2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cfa82e0f7980431eaaa017bc7fa3d6a2 2023-04-16 20:55:53,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/61b58583d9a748c7abe91945b9e5ffd5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/61b58583d9a748c7abe91945b9e5ffd5 2023-04-16 20:55:53,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/519231234abc4289967dc14adb9869e6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/519231234abc4289967dc14adb9869e6 2023-04-16 20:55:53,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e956eff68f5248949c3895ae8f437c10 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/e956eff68f5248949c3895ae8f437c10 2023-04-16 20:55:53,287 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9fd7de6c33574a6fbad5b9b7c1ceb484 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/9fd7de6c33574a6fbad5b9b7c1ceb484 2023-04-16 20:55:53,287 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6ffc7930fe3743d2a8b78507fefd6a9c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/6ffc7930fe3743d2a8b78507fefd6a9c 2023-04-16 20:55:53,288 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/457c63fe6f44435b9fb9a5bcd7bb7da5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/457c63fe6f44435b9fb9a5bcd7bb7da5 2023-04-16 20:55:53,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a3ef677afcba4b529a12b3dfcaeae53a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/a3ef677afcba4b529a12b3dfcaeae53a 2023-04-16 20:55:53,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fb5eca62a8884159a6077f463294b9d4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/fb5eca62a8884159a6077f463294b9d4 2023-04-16 20:55:53,290 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5e6652a82ecc409b916a27fd5e0e1d9d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/5e6652a82ecc409b916a27fd5e0e1d9d 2023-04-16 20:55:53,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6901af4659c94746b43b66227f27c9a8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/acf0dc1aaf8342b3b0c2f3d83d7b6221, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7de75c849671497d8277bbd8f8e1a947, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/f6f0d5c1f9894834b547ef22a616a687, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9f25557b666e4666afd2a553aeb2abc5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ddf9a7b53add4a50863d90fc894d96f6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4e1c770dde8a4e4082b42903a2aa680d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e118f4ec2f0e458eb016f161d2024508, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/8bea3d891d994f47bfdd5f056bbb7c31, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0bc86aba28c14087b3e25e3e6f294320, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/55947e04c28c4788b27f10ef359269b9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/32060b74cacd4773aae281ac9941373e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cc6125742854beea50da3626bf60361, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/65fbae91bcaa467f870179c64ab75e5a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/02b729cd9fd94943992f20b2c8128fb4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/49e1d92161d4436abb251db319cd2183, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/67082986436b4d7794ed6d6d8bbb6b3f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e9c8d4003bdc4c56a5ff782451c24dad, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff0628dea4374fa596e8542b31388efd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2daa7bcd54c74d31a9cf3fea3c9760b6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/85676f3aefd645fcb22771687f503b3d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1ae80adb6b7444f39fc1f83ed3e0aaa2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/578cdf66b13c4a658041586090f5d953, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b3a0ac95c9c414d9e9ee06673b628fe, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b197222845d4b80bae5c7aa2e529007, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/74eadf50db8444fa837094ddd435bc0c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/25c0398bb7b94b95a7abcfda517dd36e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1a861f79201548309e7edad1dc4e1a77, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/365f0e43da3c460f9d0d5c90eefbf084, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4531fdb165dc4f1abbfc18f36c70e35d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/b35efe4e9d304f3081e2542c5715d369, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/fd52954ae91d49b5b93d9f29eec90444] to archive 2023-04-16 20:55:53,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:55:53,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6901af4659c94746b43b66227f27c9a8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6901af4659c94746b43b66227f27c9a8 2023-04-16 20:55:53,295 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/acf0dc1aaf8342b3b0c2f3d83d7b6221 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/acf0dc1aaf8342b3b0c2f3d83d7b6221 2023-04-16 20:55:53,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7de75c849671497d8277bbd8f8e1a947 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/7de75c849671497d8277bbd8f8e1a947 2023-04-16 20:55:53,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/f6f0d5c1f9894834b547ef22a616a687 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/f6f0d5c1f9894834b547ef22a616a687 2023-04-16 20:55:53,297 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9f25557b666e4666afd2a553aeb2abc5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9f25557b666e4666afd2a553aeb2abc5 2023-04-16 20:55:53,298 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ddf9a7b53add4a50863d90fc894d96f6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ddf9a7b53add4a50863d90fc894d96f6 2023-04-16 20:55:53,298 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4e1c770dde8a4e4082b42903a2aa680d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4e1c770dde8a4e4082b42903a2aa680d 2023-04-16 20:55:53,299 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e118f4ec2f0e458eb016f161d2024508 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e118f4ec2f0e458eb016f161d2024508 2023-04-16 20:55:53,300 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/8bea3d891d994f47bfdd5f056bbb7c31 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/8bea3d891d994f47bfdd5f056bbb7c31 2023-04-16 20:55:53,301 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0bc86aba28c14087b3e25e3e6f294320 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/0bc86aba28c14087b3e25e3e6f294320 2023-04-16 20:55:53,301 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/55947e04c28c4788b27f10ef359269b9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/55947e04c28c4788b27f10ef359269b9 2023-04-16 20:55:53,302 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/32060b74cacd4773aae281ac9941373e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/32060b74cacd4773aae281ac9941373e 2023-04-16 20:55:53,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cc6125742854beea50da3626bf60361 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/9cc6125742854beea50da3626bf60361 2023-04-16 20:55:53,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/65fbae91bcaa467f870179c64ab75e5a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/65fbae91bcaa467f870179c64ab75e5a 2023-04-16 20:55:53,304 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/02b729cd9fd94943992f20b2c8128fb4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/02b729cd9fd94943992f20b2c8128fb4 2023-04-16 20:55:53,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/49e1d92161d4436abb251db319cd2183 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/49e1d92161d4436abb251db319cd2183 2023-04-16 20:55:53,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/67082986436b4d7794ed6d6d8bbb6b3f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/67082986436b4d7794ed6d6d8bbb6b3f 2023-04-16 20:55:53,306 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e9c8d4003bdc4c56a5ff782451c24dad to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/e9c8d4003bdc4c56a5ff782451c24dad 2023-04-16 20:55:53,307 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff0628dea4374fa596e8542b31388efd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff0628dea4374fa596e8542b31388efd 2023-04-16 20:55:53,308 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2daa7bcd54c74d31a9cf3fea3c9760b6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/2daa7bcd54c74d31a9cf3fea3c9760b6 2023-04-16 20:55:53,308 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/85676f3aefd645fcb22771687f503b3d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/85676f3aefd645fcb22771687f503b3d 2023-04-16 20:55:53,309 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1ae80adb6b7444f39fc1f83ed3e0aaa2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1ae80adb6b7444f39fc1f83ed3e0aaa2 2023-04-16 20:55:53,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/578cdf66b13c4a658041586090f5d953 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/578cdf66b13c4a658041586090f5d953 2023-04-16 20:55:53,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b3a0ac95c9c414d9e9ee06673b628fe to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b3a0ac95c9c414d9e9ee06673b628fe 2023-04-16 20:55:53,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b197222845d4b80bae5c7aa2e529007 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/6b197222845d4b80bae5c7aa2e529007 2023-04-16 20:55:53,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/74eadf50db8444fa837094ddd435bc0c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/74eadf50db8444fa837094ddd435bc0c 2023-04-16 20:55:53,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/25c0398bb7b94b95a7abcfda517dd36e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/25c0398bb7b94b95a7abcfda517dd36e 2023-04-16 20:55:53,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1a861f79201548309e7edad1dc4e1a77 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/1a861f79201548309e7edad1dc4e1a77 2023-04-16 20:55:53,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/365f0e43da3c460f9d0d5c90eefbf084 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/365f0e43da3c460f9d0d5c90eefbf084 2023-04-16 20:55:53,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4531fdb165dc4f1abbfc18f36c70e35d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/4531fdb165dc4f1abbfc18f36c70e35d 2023-04-16 20:55:53,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/b35efe4e9d304f3081e2542c5715d369 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/b35efe4e9d304f3081e2542c5715d369 2023-04-16 20:55:53,316 DEBUG [StoreCloser-TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/fd52954ae91d49b5b93d9f29eec90444 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/fd52954ae91d49b5b93d9f29eec90444 2023-04-16 20:55:53,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/recovered.edits/919.seqid, newMaxSeqId=919, maxSeqId=1 2023-04-16 20:55:53,321 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99. 2023-04-16 20:55:53,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 41b019418b53f9031c0d7e1ee3de5f99: 2023-04-16 20:55:53,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:53,322 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=41b019418b53f9031c0d7e1ee3de5f99, regionState=CLOSED 2023-04-16 20:55:53,322 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678553322"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678553322"}]},"ts":"1681678553322"} 2023-04-16 20:55:53,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-16 20:55:53,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 41b019418b53f9031c0d7e1ee3de5f99, server=jenkins-hbase4.apache.org,41211,1681678416093 in 534 msec 2023-04-16 20:55:53,325 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-16 20:55:53,325 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=41b019418b53f9031c0d7e1ee3de5f99, UNASSIGN in 538 msec 2023-04-16 20:55:53,326 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678553326"}]},"ts":"1681678553326"} 2023-04-16 20:55:53,326 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 20:55:53,328 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 20:55:53,329 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 549 msec 2023-04-16 20:55:53,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 20:55:53,385 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-16 20:55:53,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 20:55:53,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:53,387 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:53,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-16 20:55:53,388 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:53,392 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:53,394 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/recovered.edits] 2023-04-16 20:55:53,396 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/61c0f8385c7146d484236e4b8787fb7c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/61c0f8385c7146d484236e4b8787fb7c 2023-04-16 20:55:53,397 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d3772e489c4d4b5e91059c219f08edb8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/A/d3772e489c4d4b5e91059c219f08edb8 2023-04-16 20:55:53,399 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/601775089b9d4bfba83ee9ffca58d3a8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/601775089b9d4bfba83ee9ffca58d3a8 2023-04-16 20:55:53,400 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cd1d2bbfd25443ffadbf4a9a8f846bba to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/B/cd1d2bbfd25443ffadbf4a9a8f846bba 2023-04-16 20:55:53,402 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/a37532671c03403490438bac04a104b2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/a37532671c03403490438bac04a104b2 2023-04-16 20:55:53,404 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff06e8bf8574477296e88b9ef0179d53 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/C/ff06e8bf8574477296e88b9ef0179d53 2023-04-16 20:55:53,406 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/recovered.edits/919.seqid to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99/recovered.edits/919.seqid 2023-04-16 20:55:53,406 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/41b019418b53f9031c0d7e1ee3de5f99 2023-04-16 20:55:53,406 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:55:53,408 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:53,409 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 20:55:53,411 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 20:55:53,412 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:53,412 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 20:55:53,412 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681678553412"}]},"ts":"9223372036854775807"} 2023-04-16 20:55:53,413 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 20:55:53,413 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 41b019418b53f9031c0d7e1ee3de5f99, NAME => 'TestAcidGuarantees,,1681678529201.41b019418b53f9031c0d7e1ee3de5f99.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 20:55:53,413 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 20:55:53,413 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681678553413"}]},"ts":"9223372036854775807"} 2023-04-16 20:55:53,414 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 20:55:53,416 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:53,416 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 31 msec 2023-04-16 20:55:53,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-16 20:55:53,489 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-16 20:55:53,500 INFO [Listener at localhost/45179] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=236 (was 227) - Thread LEAK? -, OpenFileDescriptor=601 (was 537) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=428 (was 373) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=822 (was 395) - AvailableMemoryMB LEAK? - 2023-04-16 20:55:53,512 INFO [Listener at localhost/45179] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=236, OpenFileDescriptor=601, MaxFileDescriptor=60000, SystemLoadAverage=428, ProcessCount=169, AvailableMemoryMB=823 2023-04-16 20:55:53,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 20:55:53,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:55:53,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:53,516 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 20:55:53,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-16 20:55:53,517 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 20:55:53,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 20:55:53,518 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:53,518 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40 empty. 2023-04-16 20:55:53,518 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:53,519 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:55:53,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 20:55:53,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 20:55:53,925 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 20:55:53,926 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9853af3382d774c4b02dae6a5c604d40, NAME => 'TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp 2023-04-16 20:55:54,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 20:55:54,334 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:55:54,334 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9853af3382d774c4b02dae6a5c604d40, disabling compactions & flushes 2023-04-16 20:55:54,334 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:54,334 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:54,334 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. after waiting 0 ms 2023-04-16 20:55:54,334 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:54,334 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:54,334 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:54,336 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 20:55:54,337 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678554337"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678554337"}]},"ts":"1681678554337"} 2023-04-16 20:55:54,338 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 20:55:54,338 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 20:55:54,338 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678554338"}]},"ts":"1681678554338"} 2023-04-16 20:55:54,339 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 20:55:54,342 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, ASSIGN}] 2023-04-16 20:55:54,343 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, ASSIGN 2023-04-16 20:55:54,344 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=false 2023-04-16 20:55:54,495 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=9853af3382d774c4b02dae6a5c604d40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:54,495 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678554495"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678554495"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678554495"}]},"ts":"1681678554495"} 2023-04-16 20:55:54,496 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:55:54,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 20:55:54,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:54,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9853af3382d774c4b02dae6a5c604d40, NAME => 'TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:55:54,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:54,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:55:54,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:54,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:54,652 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:54,653 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A 2023-04-16 20:55:54,653 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A 2023-04-16 20:55:54,653 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:54,654 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9853af3382d774c4b02dae6a5c604d40 columnFamilyName A 2023-04-16 20:55:54,654 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(310): Store=9853af3382d774c4b02dae6a5c604d40/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:54,654 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:54,655 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B 2023-04-16 20:55:54,655 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B 2023-04-16 20:55:54,655 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:54,655 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9853af3382d774c4b02dae6a5c604d40 columnFamilyName B 2023-04-16 20:55:54,656 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(310): Store=9853af3382d774c4b02dae6a5c604d40/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:54,656 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:54,657 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C 2023-04-16 20:55:54,657 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C 2023-04-16 20:55:54,657 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:54,657 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9853af3382d774c4b02dae6a5c604d40 columnFamilyName C 2023-04-16 20:55:54,657 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(310): Store=9853af3382d774c4b02dae6a5c604d40/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:54,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:54,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:54,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:54,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:55:54,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:54,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 20:55:54,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9853af3382d774c4b02dae6a5c604d40; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60176910, jitterRate=-0.10329416394233704}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:55:54,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:54,663 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., pid=56, masterSystemTime=1681678554648 2023-04-16 20:55:54,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:54,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:54,664 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=9853af3382d774c4b02dae6a5c604d40, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:54,664 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678554664"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678554664"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678554664"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678554664"}]},"ts":"1681678554664"} 2023-04-16 20:55:54,666 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-16 20:55:54,666 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 in 169 msec 2023-04-16 20:55:54,667 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-16 20:55:54,667 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, ASSIGN in 324 msec 2023-04-16 20:55:54,668 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 20:55:54,668 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678554668"}]},"ts":"1681678554668"} 2023-04-16 20:55:54,669 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 20:55:54,671 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 20:55:54,671 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-04-16 20:55:55,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 20:55:55,622 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-16 20:55:55,622 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:55,629 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:55,630 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:55,631 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40688, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:55,632 DEBUG [Listener at localhost/45179] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 20:55:55,634 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36590, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 20:55:55,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 20:55:55,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 20:55:55,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-16 20:55:55,648 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 20:55:55,649 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 20:55:55,649 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 20:55:55,651 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-16 20:55:55,653 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, REOPEN/MOVE}] 2023-04-16 20:55:55,653 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, REOPEN/MOVE 2023-04-16 20:55:55,654 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=9853af3382d774c4b02dae6a5c604d40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:55,654 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678555654"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678555654"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678555654"}]},"ts":"1681678555654"} 2023-04-16 20:55:55,655 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:55:55,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:55,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9853af3382d774c4b02dae6a5c604d40, disabling compactions & flushes 2023-04-16 20:55:55,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:55,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:55,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. after waiting 0 ms 2023-04-16 20:55:55,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:55,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-16 20:55:55,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:55,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:55,810 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 9853af3382d774c4b02dae6a5c604d40 to self. 2023-04-16 20:55:55,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:55,812 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=9853af3382d774c4b02dae6a5c604d40, regionState=CLOSED 2023-04-16 20:55:55,812 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678555812"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678555812"}]},"ts":"1681678555812"} 2023-04-16 20:55:55,814 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-16 20:55:55,814 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 in 158 msec 2023-04-16 20:55:55,814 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,41211,1681678416093; forceNewPlan=false, retain=true 2023-04-16 20:55:55,964 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=9853af3382d774c4b02dae6a5c604d40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:55,965 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678555964"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678555964"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678555964"}]},"ts":"1681678555964"} 2023-04-16 20:55:55,966 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:55:56,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9853af3382d774c4b02dae6a5c604d40, NAME => 'TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.', STARTKEY => '', ENDKEY => ''} 2023-04-16 20:55:56,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 20:55:56,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,121 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,122 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A 2023-04-16 20:55:56,122 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A 2023-04-16 20:55:56,122 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:56,123 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9853af3382d774c4b02dae6a5c604d40 columnFamilyName A 2023-04-16 20:55:56,124 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(310): Store=9853af3382d774c4b02dae6a5c604d40/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:56,124 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,125 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B 2023-04-16 20:55:56,125 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B 2023-04-16 20:55:56,125 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:56,125 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9853af3382d774c4b02dae6a5c604d40 columnFamilyName B 2023-04-16 20:55:56,125 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(310): Store=9853af3382d774c4b02dae6a5c604d40/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:56,125 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,126 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C 2023-04-16 20:55:56,126 DEBUG [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C 2023-04-16 20:55:56,126 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 20:55:56,126 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9853af3382d774c4b02dae6a5c604d40 columnFamilyName C 2023-04-16 20:55:56,126 INFO [StoreOpener-9853af3382d774c4b02dae6a5c604d40-1] regionserver.HStore(310): Store=9853af3382d774c4b02dae6a5c604d40/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 20:55:56,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 20:55:56,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9853af3382d774c4b02dae6a5c604d40; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69336030, jitterRate=0.03318735957145691}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 20:55:56,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:56,132 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., pid=61, masterSystemTime=1681678556117 2023-04-16 20:55:56,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,133 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=9853af3382d774c4b02dae6a5c604d40, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,134 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678556133"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681678556133"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681678556133"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681678556133"}]},"ts":"1681678556133"} 2023-04-16 20:55:56,135 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-16 20:55:56,135 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 in 168 msec 2023-04-16 20:55:56,136 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-16 20:55:56,136 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, REOPEN/MOVE in 482 msec 2023-04-16 20:55:56,138 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-16 20:55:56,138 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 486 msec 2023-04-16 20:55:56,139 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 502 msec 2023-04-16 20:55:56,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-16 20:55:56,140 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,145 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,145 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,149 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,149 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,153 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,153 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,158 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,158 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,162 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,162 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,168 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,168 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,172 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,172 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x0eedd898 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,180 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@308b3f39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,180 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x618c0022 to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,188 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1453d93, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,189 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(139): Connect 0x1d33872d to 127.0.0.1:54578 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 20:55:56,192 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1cbf09b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 20:55:56,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:56,204 DEBUG [hconnection-0x561f5bb5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,204 DEBUG [hconnection-0x3310d60b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,212 DEBUG [hconnection-0x7fe15115-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,212 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40704, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,212 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,212 DEBUG [hconnection-0x55bc1a78-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,213 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40724, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,218 DEBUG [hconnection-0xe25ee13-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,218 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40740, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,220 DEBUG [hconnection-0x15eb42c4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,220 DEBUG [hconnection-0xba38536-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,220 DEBUG [hconnection-0x6cb0f38d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,220 DEBUG [hconnection-0x4f785d1e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,223 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40750, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,223 DEBUG [hconnection-0xc286830-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 20:55:56,224 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40754, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,224 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,225 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40748, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,225 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40762, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,227 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 20:55:56,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:56,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:56,240 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:56,240 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,240 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:56,240 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:56,241 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,241 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,242 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,243 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:56,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:56,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:55:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:56,249 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,249 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:56,249 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:56,249 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:56,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:55:56,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:56,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:55:56,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:56,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:55:56,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,252 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:56,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10e8b02c[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:56,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:56,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,255 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:56,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,255 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:56,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:56,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:56,260 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:56,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:56,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:56,260 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:56,260 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:56,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,263 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678616263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678616265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678616267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678556262, max timestamp=1681678556265Num uniques 859; 2023-04-16 20:55:56,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 20:55:56,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678616267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678556262, max timestamp=1681678556265Num uniques 859; 2023-04-16 20:55:56,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 20:55:56,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678556262, max timestamp=1681678556265Num uniques 859; 2023-04-16 20:55:56,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 20:55:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,272 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,276 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,276 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:56,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:56,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:56,277 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,277 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:56,277 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:56,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,286 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416576fa48476cd40d08dc98e6a8f220dcd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416576fa48476cd40d08dc98e6a8f220dcd 2023-04-16 20:55:56,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678616323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678616369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678616372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678616373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678616373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,377 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:56,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:56,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:56,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:56,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:56,380 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:56,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:56,380 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:56,381 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,381 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,382 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,382 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,382 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,382 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:56,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:56,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:56,383 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:56,383 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,383 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:56,383 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:56,383 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:56,383 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,383 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:56,383 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:56,384 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:56,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:56,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:56,384 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:56,388 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:56,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eff48d6[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:56,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:56,390 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,392 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:56,393 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,393 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:56,394 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:56,396 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:56,396 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:56,396 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:56,396 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:56,396 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:56,397 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,398 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,398 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,401 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:56,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:56,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,402 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:56,402 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:56,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:56,404 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,404 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678616425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,502 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:56,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:56,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:56,505 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:56,506 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,506 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:56,506 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:56,506 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,506 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,508 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:56,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:56,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:56,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:56,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:56,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:56,510 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:56,512 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,512 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,512 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,512 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:56,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263d858d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:56,514 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:56,514 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:56,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:56,516 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:56,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:56,518 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:56,518 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:56,518 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:56,518 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:56,518 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:56,520 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,520 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,520 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,521 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,522 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,524 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,525 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:56,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:56,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,525 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:56,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,525 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:56,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678616573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678616576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678616577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678616577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,625 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:56,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:56,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:56,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:56,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:56,629 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:56,629 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,629 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:56,629 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:56,629 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,629 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678616629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,631 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,631 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,631 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,631 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:56,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:56,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:56,632 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,632 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:56,632 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:56,632 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,633 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:56,633 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:56,633 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,633 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:56,633 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:56,634 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,634 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,634 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,634 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:56,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d46ebbf[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:56,636 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:56,636 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,638 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,638 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:56,638 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:56,638 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:56,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,638 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:56,640 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,640 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,640 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,640 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:56,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:56,642 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:56,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:56,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:56,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:56,642 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:56,644 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,644 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,644 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:56,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:56,649 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,649 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,649 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:56,650 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:56,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,695 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/2704a0642d9a46e9a8c213f04e042334 2023-04-16 20:55:56,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/bb9b03a0022e455692b1d3ad78331d43 2023-04-16 20:55:56,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/94f436f7162c49a5a17d38d413a49cef 2023-04-16 20:55:56,718 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/2704a0642d9a46e9a8c213f04e042334 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2704a0642d9a46e9a8c213f04e042334 2023-04-16 20:55:56,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2704a0642d9a46e9a8c213f04e042334, entries=150, sequenceid=17, filesize=25.1 K 2023-04-16 20:55:56,722 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/bb9b03a0022e455692b1d3ad78331d43 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb9b03a0022e455692b1d3ad78331d43 2023-04-16 20:55:56,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb9b03a0022e455692b1d3ad78331d43, entries=150, sequenceid=17, filesize=11.6 K 2023-04-16 20:55:56,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/94f436f7162c49a5a17d38d413a49cef as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/94f436f7162c49a5a17d38d413a49cef 2023-04-16 20:55:56,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/94f436f7162c49a5a17d38d413a49cef, entries=150, sequenceid=17, filesize=11.6 K 2023-04-16 20:55:56,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=112.87 KB/115575 for 9853af3382d774c4b02dae6a5c604d40 in 481ms, sequenceid=17, compaction requested=false 2023-04-16 20:55:56,729 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-16 20:55:56,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:56,750 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:56,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:56,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:56,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:56,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:56,753 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:56,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:56,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:56,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,756 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:56,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:56,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:56,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:56,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:56,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:56,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:56,758 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:56,758 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,758 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:56,758 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=112.87 KB heapSize=322.48 KB 2023-04-16 20:55:56,758 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:55:56,758 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:56,758 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:55:56,758 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:56,758 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:55:56,758 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:56,773 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416aefba12179934ab2b49c33fce3c35cc5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416aefba12179934ab2b49c33fce3c35cc5 2023-04-16 20:55:56,786 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=37.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/e502e329789a42219dd2453516f8edf3 2023-04-16 20:55:56,796 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/65eb332deac942bfafb96a2ea62a26f3 2023-04-16 20:55:56,807 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/f54324efdf8b4c6b9fb9bb0148380138 2023-04-16 20:55:56,815 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/e502e329789a42219dd2453516f8edf3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e502e329789a42219dd2453516f8edf3 2023-04-16 20:55:56,818 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e502e329789a42219dd2453516f8edf3, entries=150, sequenceid=51, filesize=25.1 K 2023-04-16 20:55:56,819 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/65eb332deac942bfafb96a2ea62a26f3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/65eb332deac942bfafb96a2ea62a26f3 2023-04-16 20:55:56,822 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/65eb332deac942bfafb96a2ea62a26f3, entries=150, sequenceid=51, filesize=11.6 K 2023-04-16 20:55:56,823 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/f54324efdf8b4c6b9fb9bb0148380138 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f54324efdf8b4c6b9fb9bb0148380138 2023-04-16 20:55:56,826 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f54324efdf8b4c6b9fb9bb0148380138, entries=150, sequenceid=51, filesize=11.6 K 2023-04-16 20:55:56,827 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~112.87 KB/115575, heapSize ~322.43 KB/330168, currentSize=0 B/0 for 9853af3382d774c4b02dae6a5c604d40 in 69ms, sequenceid=51, compaction requested=false 2023-04-16 20:55:56,828 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:56,828 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,830 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:56,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@787a47eb[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:56,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:56,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,836 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:56,836 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,836 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:56,839 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:56,842 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:56,842 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:56,842 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:56,842 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:56,842 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:56,844 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,844 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,844 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,845 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,848 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:56,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:56,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:56,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,848 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,848 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:56,848 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:56,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:56,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:55:56,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:55:56,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:56,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:55:56,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:56,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:55:56,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:56,935 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416b98e0f4c1e7e4c8ebc462be10a8119c6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b98e0f4c1e7e4c8ebc462be10a8119c6 2023-04-16 20:55:56,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678556930, max timestamp=1681678556940Num uniques 859; 2023-04-16 20:55:56,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:56,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678556930, max timestamp=1681678556940Num uniques 859; 2023-04-16 20:55:56,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:56,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678556930, max timestamp=1681678556940Num uniques 859; 2023-04-16 20:55:56,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:56,949 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:56,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:56,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:56,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:56,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:56,957 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:56,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:56,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:56,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,958 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,958 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,959 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:56,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678616955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678616955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678616956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:56,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:56,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:56,961 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,961 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:56,961 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:56,961 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:56,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678616958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:56,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678616959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:56,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ca719ca[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:56,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:56,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,967 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:56,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:56,967 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:56,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:56,968 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,968 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:56,968 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,968 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:56,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:56,970 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:56,970 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:56,971 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:56,971 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:56,971 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:56,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:56,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:56,974 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,975 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:56,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,979 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:56,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:56,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:56,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:56,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:56,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:56,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:56,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:56,979 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:56,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:56,979 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:56,979 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:57,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678617060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678617061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678617061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678617063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678617063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,079 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:57,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:57,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:57,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:57,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:57,083 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:57,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:57,083 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:57,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,084 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,085 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,086 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,086 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:57,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:57,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:57,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:57,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:57,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:57,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:57,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:57,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:57,087 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,087 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:57,089 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,089 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fcf63e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:57,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:57,092 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,093 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,093 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:57,093 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:57,093 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:57,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,094 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:57,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:57,096 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,096 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:57,098 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:57,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:57,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:57,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:57,098 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:57,100 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,101 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,101 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,101 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,102 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,105 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:57,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:57,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:57,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,105 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:57,105 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:57,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,105 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,206 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:57,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:57,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:57,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:57,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:57,209 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:57,209 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,209 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:57,209 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:57,210 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,210 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,211 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,211 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:57,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:57,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:57,213 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,214 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:57,214 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:57,214 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,214 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:57,214 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:57,214 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,214 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:57,214 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:57,216 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,216 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,216 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,216 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c7de44d[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:57,218 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:57,218 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,220 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:57,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:57,220 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:57,221 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,221 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:57,222 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,222 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:57,222 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,222 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:57,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:57,225 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:57,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:57,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:57,225 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:57,225 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:57,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,230 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,231 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,233 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,235 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:57,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:57,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,236 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,236 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,236 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:57,236 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:57,236 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,236 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678617264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678617264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678617264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678617266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678617266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,336 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:57,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:57,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:57,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:57,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:57,339 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:57,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:57,340 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:57,340 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,340 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,341 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,341 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,342 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,342 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:57,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:57,343 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,343 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:57,343 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:57,343 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:57,345 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6612b736[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:57,346 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:57,347 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,353 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:57,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:57,353 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:57,354 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,354 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:57,355 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,355 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:57,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:57,357 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:57,357 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:57,357 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:57,357 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:57,357 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:57,358 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/32900ffa448042b096c346e1c8a87a79 2023-04-16 20:55:57,359 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,360 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,364 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:57,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:57,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,367 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,367 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:57,367 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,368 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/0068cd5599604468a173dc0e23a79290 2023-04-16 20:55:57,376 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/47a95e71c90e4590aa6ef549a2f055ee 2023-04-16 20:55:57,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/32900ffa448042b096c346e1c8a87a79 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/32900ffa448042b096c346e1c8a87a79 2023-04-16 20:55:57,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/32900ffa448042b096c346e1c8a87a79, entries=400, sequenceid=62, filesize=59.0 K 2023-04-16 20:55:57,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/0068cd5599604468a173dc0e23a79290 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0068cd5599604468a173dc0e23a79290 2023-04-16 20:55:57,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0068cd5599604468a173dc0e23a79290, entries=150, sequenceid=62, filesize=11.6 K 2023-04-16 20:55:57,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/47a95e71c90e4590aa6ef549a2f055ee as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/47a95e71c90e4590aa6ef549a2f055ee 2023-04-16 20:55:57,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/47a95e71c90e4590aa6ef549a2f055ee, entries=150, sequenceid=62, filesize=11.6 K 2023-04-16 20:55:57,395 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9853af3382d774c4b02dae6a5c604d40 in 489ms, sequenceid=62, compaction requested=true 2023-04-16 20:55:57,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:57,396 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:57,396 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:57,396 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:57,396 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:57,396 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:57,397 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 111946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:57,397 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:55:57,397 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:57,397 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,398 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:55:57,398 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2704a0642d9a46e9a8c213f04e042334, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e502e329789a42219dd2453516f8edf3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/32900ffa448042b096c346e1c8a87a79] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=109.3 K 2023-04-16 20:55:57,398 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,398 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb9b03a0022e455692b1d3ad78331d43, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/65eb332deac942bfafb96a2ea62a26f3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0068cd5599604468a173dc0e23a79290] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=34.8 K 2023-04-16 20:55:57,398 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2704a0642d9a46e9a8c213f04e042334, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681678556242 2023-04-16 20:55:57,398 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting bb9b03a0022e455692b1d3ad78331d43, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681678556242 2023-04-16 20:55:57,398 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e502e329789a42219dd2453516f8edf3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681678556296 2023-04-16 20:55:57,398 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 65eb332deac942bfafb96a2ea62a26f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681678556296 2023-04-16 20:55:57,399 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 32900ffa448042b096c346e1c8a87a79, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678556887 2023-04-16 20:55:57,399 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0068cd5599604468a173dc0e23a79290, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678556896 2023-04-16 20:55:57,406 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:57,407 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#554 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:57,467 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:57,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:57,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:57,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:57,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:57,471 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:57,471 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,471 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:57,471 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:57,471 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,471 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,473 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,473 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,474 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:57,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:57,475 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,475 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:57,475 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-16 20:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:55:57,476 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:57,493 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304167c1c6d3d786642959d643f170f59f5e6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167c1c6d3d786642959d643f170f59f5e6 2023-04-16 20:55:57,499 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/3aa158af4c21488e9c6af541fa26f2b6 2023-04-16 20:55:57,509 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/0819f206a4a64dfd95401f73ef8e575a 2023-04-16 20:55:57,520 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/501a94e96ee341e8b259f4098faf7156 2023-04-16 20:55:57,523 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/3aa158af4c21488e9c6af541fa26f2b6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3aa158af4c21488e9c6af541fa26f2b6 2023-04-16 20:55:57,526 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3aa158af4c21488e9c6af541fa26f2b6, entries=150, sequenceid=97, filesize=25.1 K 2023-04-16 20:55:57,526 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/0819f206a4a64dfd95401f73ef8e575a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0819f206a4a64dfd95401f73ef8e575a 2023-04-16 20:55:57,530 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0819f206a4a64dfd95401f73ef8e575a, entries=150, sequenceid=97, filesize=11.6 K 2023-04-16 20:55:57,530 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/501a94e96ee341e8b259f4098faf7156 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/501a94e96ee341e8b259f4098faf7156 2023-04-16 20:55:57,537 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/501a94e96ee341e8b259f4098faf7156, entries=150, sequenceid=97, filesize=11.6 K 2023-04-16 20:55:57,540 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 9853af3382d774c4b02dae6a5c604d40 in 65ms, sequenceid=97, compaction requested=true 2023-04-16 20:55:57,540 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:57,540 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:57,541 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:57,542 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,542 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,543 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,543 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:57,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42df9b77[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:57,545 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:57,545 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,546 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:57,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:57,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:57,546 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,546 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:57,547 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,547 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,547 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,547 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:57,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:57,551 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:57,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:57,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:57,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:57,551 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:57,558 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,558 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,558 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,559 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,560 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:57,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:57,562 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:57,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,562 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,563 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:57,563 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:57,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:57,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:55:57,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:55:57,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:57,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:55:57,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:57,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:55:57,583 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:57,594 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416efce844fccce4b1da3007f92320cc534 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416efce844fccce4b1da3007f92320cc534 2023-04-16 20:55:57,601 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/ec3bba95fab74452a9e1249d94deeff0 2023-04-16 20:55:57,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681678557597, max timestamp=1681678557607Num uniques 859; 2023-04-16 20:55:57,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 20:55:57,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678557597, max timestamp=1681678557607Num uniques 859; 2023-04-16 20:55:57,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:57,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678557597, max timestamp=1681678557608Num uniques 859; 2023-04-16 20:55:57,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:55:57,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678617657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678617658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678617659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678617659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,663 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:57,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678617660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:57,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:57,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:57,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:57,667 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:57,667 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,667 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:57,667 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:57,667 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,667 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,669 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:57,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:57,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:57,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:57,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:57,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,673 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:57,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:57,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:57,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:57,679 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,679 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a3c814[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:57,682 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:57,682 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,685 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:57,685 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,686 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:57,687 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:57,690 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:57,690 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:57,690 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:57,690 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:57,690 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:57,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,692 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,693 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,694 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:57,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:57,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,699 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,700 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:57,700 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:57,699 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,700 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,700 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,700 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:57,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678617762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678617762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678617763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678617763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678617764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,800 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:57,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:57,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:57,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:57,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:57,803 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:57,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:57,803 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:57,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,805 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,805 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,805 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,805 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:57,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:57,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:57,807 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,807 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:57,807 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:57,807 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:57,809 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@505f05a1[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:57,811 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:57,811 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,812 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:57,812 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,812 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:57,815 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:57,817 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:57,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:57,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:57,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:57,817 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:57,819 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,819 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,819 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,820 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,821 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,823 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,823 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,823 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,823 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:57,823 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:57,823 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,824 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,824 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,824 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,824 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,824 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:57,824 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:57,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,831 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/1f527b07382e410bb8ba2f290ed063fc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1f527b07382e410bb8ba2f290ed063fc 2023-04-16 20:55:57,835 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into 1f527b07382e410bb8ba2f290ed063fc(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:57,835 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:57,835 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=13, startTime=1681678557396; duration=0sec 2023-04-16 20:55:57,835 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:57,835 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:57,836 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:57,836 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/C is initiating minor compaction (all files) 2023-04-16 20:55:57,836 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/C in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,837 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/94f436f7162c49a5a17d38d413a49cef, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f54324efdf8b4c6b9fb9bb0148380138, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/47a95e71c90e4590aa6ef549a2f055ee, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/501a94e96ee341e8b259f4098faf7156] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=46.4 K 2023-04-16 20:55:57,837 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 94f436f7162c49a5a17d38d413a49cef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681678556242 2023-04-16 20:55:57,838 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting f54324efdf8b4c6b9fb9bb0148380138, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681678556296 2023-04-16 20:55:57,838 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 47a95e71c90e4590aa6ef549a2f055ee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678556896 2023-04-16 20:55:57,838 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 501a94e96ee341e8b259f4098faf7156, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678556955 2023-04-16 20:55:57,845 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#C#compaction#561 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:57,847 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/2675171c9ed7406e8fb4184926925ac2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2675171c9ed7406e8fb4184926925ac2 2023-04-16 20:55:57,852 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into 2675171c9ed7406e8fb4184926925ac2(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:57,852 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:57,852 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=13, startTime=1681678557396; duration=0sec 2023-04-16 20:55:57,852 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:57,855 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/994f25c3f5364a64803c422daec6f35c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/994f25c3f5364a64803c422daec6f35c 2023-04-16 20:55:57,859 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/C of 9853af3382d774c4b02dae6a5c604d40 into 994f25c3f5364a64803c422daec6f35c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:57,859 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:57,859 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/C, priority=12, startTime=1681678557396; duration=0sec 2023-04-16 20:55:57,860 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:57,924 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:57,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:57,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:57,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:57,927 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:57,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:57,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:57,928 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,928 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,929 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:57,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:57,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:57,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:57,931 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,931 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:57,931 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:57,931 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:57,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:57,934 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2307a39b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:57,936 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:57,936 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,937 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:57,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:57,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:57,938 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,938 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,939 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:57,942 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:57,942 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:57,942 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:57,942 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:57,942 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:57,944 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,944 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,944 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:57,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:57,946 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:57,947 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,950 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,951 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,951 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:57,951 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:57,951 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:57,951 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:57,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:57,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:57,951 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:57,951 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:57,951 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:57,951 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:57,951 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:57,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:57,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:57,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678617966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678617966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678617967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678617967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:57,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:57,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678617968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,017 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/5d5e53bf582f4c42ac3b99fc4402cd80 2023-04-16 20:55:58,030 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/1ebb97974a034fe390da0f76071a4d4c 2023-04-16 20:55:58,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/ec3bba95fab74452a9e1249d94deeff0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ec3bba95fab74452a9e1249d94deeff0 2023-04-16 20:55:58,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ec3bba95fab74452a9e1249d94deeff0, entries=400, sequenceid=108, filesize=59.0 K 2023-04-16 20:55:58,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/5d5e53bf582f4c42ac3b99fc4402cd80 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5d5e53bf582f4c42ac3b99fc4402cd80 2023-04-16 20:55:58,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5d5e53bf582f4c42ac3b99fc4402cd80, entries=150, sequenceid=108, filesize=11.6 K 2023-04-16 20:55:58,041 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/1ebb97974a034fe390da0f76071a4d4c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/1ebb97974a034fe390da0f76071a4d4c 2023-04-16 20:55:58,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/1ebb97974a034fe390da0f76071a4d4c, entries=150, sequenceid=108, filesize=11.6 K 2023-04-16 20:55:58,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.94 KB/135110 for 9853af3382d774c4b02dae6a5c604d40 in 464ms, sequenceid=108, compaction requested=true 2023-04-16 20:55:58,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:58,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:58,045 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:58,045 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:58,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:58,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:58,046 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 112049 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:58,046 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:55:58,046 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:58,046 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,046 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:55:58,046 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2675171c9ed7406e8fb4184926925ac2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3aa158af4c21488e9c6af541fa26f2b6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ec3bba95fab74452a9e1249d94deeff0] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=109.4 K 2023-04-16 20:55:58,046 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,047 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1f527b07382e410bb8ba2f290ed063fc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0819f206a4a64dfd95401f73ef8e575a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5d5e53bf582f4c42ac3b99fc4402cd80] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=34.9 K 2023-04-16 20:55:58,047 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2675171c9ed7406e8fb4184926925ac2, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678556896 2023-04-16 20:55:58,047 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1f527b07382e410bb8ba2f290ed063fc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681678556896 2023-04-16 20:55:58,047 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3aa158af4c21488e9c6af541fa26f2b6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678556955 2023-04-16 20:55:58,047 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0819f206a4a64dfd95401f73ef8e575a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678556955 2023-04-16 20:55:58,048 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ec3bba95fab74452a9e1249d94deeff0, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681678557571 2023-04-16 20:55:58,048 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5d5e53bf582f4c42ac3b99fc4402cd80, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681678557580 2023-04-16 20:55:58,051 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:58,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:58,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:58,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:58,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:58,055 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:58,055 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:58,055 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:58,055 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:58,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,056 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,056 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#564 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:58,057 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:58,057 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,058 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,058 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:58,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:58,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:58,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:58,064 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:58,064 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:58,064 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:58,069 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#563 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:58,072 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,072 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:58,072 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=131.94 KB heapSize=370.53 KB 2023-04-16 20:55:58,072 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:55:58,073 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:58,073 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:55:58,073 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:58,073 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:55:58,073 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:55:58,089 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/1db593803dce49a28a57aad6e6892118 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1db593803dce49a28a57aad6e6892118 2023-04-16 20:55:58,098 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into 1db593803dce49a28a57aad6e6892118(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:58,098 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:58,098 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=13, startTime=1681678558045; duration=0sec 2023-04-16 20:55:58,098 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:58,098 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-16 20:55:58,099 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-16 20:55:58,099 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-16 20:55:58,099 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. because compaction request was cancelled 2023-04-16 20:55:58,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:58,271 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:58,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678618275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678618276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678618277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678618279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678618279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678618380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678618380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678618380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678618384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678618384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,501 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416554d6af26b4746eb9df50a44c3611073 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416554d6af26b4746eb9df50a44c3611073 2023-04-16 20:55:58,506 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/9d9e036edd414faf93955a3bffb5d23a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/9d9e036edd414faf93955a3bffb5d23a 2023-04-16 20:55:58,507 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/e6abc28f59344518bf95b9ec6f27066b 2023-04-16 20:55:58,512 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into 9d9e036edd414faf93955a3bffb5d23a(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:58,512 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:58,512 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=13, startTime=1681678558045; duration=0sec 2023-04-16 20:55:58,512 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:58,521 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/62963c595aef48e7b1ff4bc1554f6221 2023-04-16 20:55:58,530 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/f9c588a5dae04dfc8e144593068382a2 2023-04-16 20:55:58,533 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/e6abc28f59344518bf95b9ec6f27066b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e6abc28f59344518bf95b9ec6f27066b 2023-04-16 20:55:58,536 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e6abc28f59344518bf95b9ec6f27066b, entries=150, sequenceid=146, filesize=25.3 K 2023-04-16 20:55:58,537 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/62963c595aef48e7b1ff4bc1554f6221 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/62963c595aef48e7b1ff4bc1554f6221 2023-04-16 20:55:58,540 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/62963c595aef48e7b1ff4bc1554f6221, entries=150, sequenceid=146, filesize=11.8 K 2023-04-16 20:55:58,541 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/f9c588a5dae04dfc8e144593068382a2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f9c588a5dae04dfc8e144593068382a2 2023-04-16 20:55:58,544 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f9c588a5dae04dfc8e144593068382a2, entries=150, sequenceid=146, filesize=11.8 K 2023-04-16 20:55:58,544 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~370.48 KB/379376, currentSize=60.38 KB/61830 for 9853af3382d774c4b02dae6a5c604d40 in 472ms, sequenceid=146, compaction requested=true 2023-04-16 20:55:58,544 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:58,544 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,545 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:58,545 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:58,546 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:58,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3d90af[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:58,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:58,549 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,550 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:58,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:58,550 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:58,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,551 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:58,552 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:58,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:58,555 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:58,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:58,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:58,555 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:58,555 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:58,557 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,557 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:58,557 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:58,558 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,560 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:58,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:58,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:58,565 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:58,565 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:58,565 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:58,565 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:58,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:58,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:58,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:58,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:58,586 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 20:55:58,587 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:55:58,587 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:58,587 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:55:58,587 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:58,587 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:55:58,587 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:58,612 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304163b92e7a295124d639dfa2dd219bc3621 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163b92e7a295124d639dfa2dd219bc3621 2023-04-16 20:55:58,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678618617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678618617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678618618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678618620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678618620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,665 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:58,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:58,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:58,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:58,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:58,669 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:58,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:58,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:58,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:58,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,674 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:58,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:58,675 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,675 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:58,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:58,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:58,676 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,676 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:58,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:58,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:58,678 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,678 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:58,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70330874[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:58,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:58,681 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,682 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:58,682 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,682 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:58,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:58,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:58,687 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:58,687 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:58,687 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:58,687 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:58,687 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:58,690 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,690 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:58,690 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:58,691 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,693 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:58,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:58,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:58,695 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:58,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:58,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:58,696 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:58,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:58,695 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,696 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,696 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,696 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,696 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,696 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:58,696 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:58,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678618720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678618720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678618720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678618722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678618722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,796 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:58,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:58,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:58,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:58,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:58,799 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:58,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:58,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:58,799 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:58,800 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,800 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,801 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:58,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:58,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,801 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:58,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:58,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:58,802 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,802 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:58,802 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:58,803 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:58,804 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e979de[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:58,805 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:58,805 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,806 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:58,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:58,807 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:58,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,807 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:58,813 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:58,815 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:58,815 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:58,815 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:58,815 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:58,815 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:58,816 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,817 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,817 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:58,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,817 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,818 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,820 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:58,820 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:58,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:58,820 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,821 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,821 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,821 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,821 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,821 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,821 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,821 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:58,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:58,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:58,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:58,821 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:58,821 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:58,850 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 20:55:58,921 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:58,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:58,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678618921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678618922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678618922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:58,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:58,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:58,925 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:58,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:58,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:58,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:58,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,925 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,927 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:58,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:58,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:58,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:58,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:58,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678618926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:58,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:58,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678618926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:58,929 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:58,929 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:58,929 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:58,929 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:58,932 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,932 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,933 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,933 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:58,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@466c0223[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:58,934 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:58,934 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,935 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:58,935 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,935 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:58,937 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:58,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:58,937 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:58,939 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:58,939 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:58,939 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:58,939 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:58,939 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:58,941 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,941 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:58,941 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:58,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:58,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:58,942 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,944 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:58,947 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:58,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:58,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:58,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:58,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:58,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:58,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:58,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:58,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:58,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:58,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:58,948 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:58,948 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:59,017 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=162, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/8622eb9a357948d6ac0b3d14b7ee0bff 2023-04-16 20:55:59,030 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/30283da273894fea835a386d5c3976e1 2023-04-16 20:55:59,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/2546a18582bb4ac79e574f828b9263fb 2023-04-16 20:55:59,044 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/8622eb9a357948d6ac0b3d14b7ee0bff as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8622eb9a357948d6ac0b3d14b7ee0bff 2023-04-16 20:55:59,048 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:59,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8622eb9a357948d6ac0b3d14b7ee0bff, entries=200, sequenceid=162, filesize=32.1 K 2023-04-16 20:55:59,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:59,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/30283da273894fea835a386d5c3976e1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/30283da273894fea835a386d5c3976e1 2023-04-16 20:55:59,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:59,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:59,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:59,051 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:59,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:59,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:59,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:59,052 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,052 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/30283da273894fea835a386d5c3976e1, entries=150, sequenceid=162, filesize=11.8 K 2023-04-16 20:55:59,053 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:59,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:59,053 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/2546a18582bb4ac79e574f828b9263fb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2546a18582bb4ac79e574f828b9263fb 2023-04-16 20:55:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,054 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,054 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:59,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:59,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:59,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:59,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:59,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:59,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:59,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:59,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:59,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:59,055 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:59,055 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:59,055 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:59,055 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:59,055 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:59,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2546a18582bb4ac79e574f828b9263fb, entries=150, sequenceid=162, filesize=11.8 K 2023-04-16 20:55:59,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 9853af3382d774c4b02dae6a5c604d40 in 472ms, sequenceid=162, compaction requested=true 2023-04-16 20:55:59,059 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:59,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:59,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:55:59,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-16 20:55:59,059 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 20:55:59,059 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:59,059 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:59,059 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:59,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,060 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:59,060 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84734 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:59,060 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:55:59,061 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,061 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 20:55:59,061 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/9d9e036edd414faf93955a3bffb5d23a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e6abc28f59344518bf95b9ec6f27066b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8622eb9a357948d6ac0b3d14b7ee0bff] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=82.7 K 2023-04-16 20:55:59,061 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/C is initiating minor compaction (all files) 2023-04-16 20:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,061 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/C in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:59,061 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/994f25c3f5364a64803c422daec6f35c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/1ebb97974a034fe390da0f76071a4d4c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f9c588a5dae04dfc8e144593068382a2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2546a18582bb4ac79e574f828b9263fb] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=46.9 K 2023-04-16 20:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:59,061 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9d9e036edd414faf93955a3bffb5d23a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681678557580 2023-04-16 20:55:59,062 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 994f25c3f5364a64803c422daec6f35c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681678556955 2023-04-16 20:55:59,062 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e6abc28f59344518bf95b9ec6f27066b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681678557657 2023-04-16 20:55:59,062 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1ebb97974a034fe390da0f76071a4d4c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681678557580 2023-04-16 20:55:59,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:59,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a55aa9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:59,062 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8622eb9a357948d6ac0b3d14b7ee0bff, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681678558278 2023-04-16 20:55:59,062 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:59,062 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,063 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting f9c588a5dae04dfc8e144593068382a2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681678557657 2023-04-16 20:55:59,063 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2546a18582bb4ac79e574f828b9263fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681678558278 2023-04-16 20:55:59,064 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:59,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,067 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:59,067 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:59,067 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:59,067 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:59,067 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:59,067 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:59,067 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:59,069 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,069 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:59,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,070 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:59,071 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:59,073 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,076 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:59,076 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:59,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:59,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:59,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:59,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:59,076 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,076 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,076 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,076 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,076 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,076 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,077 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,077 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:59,077 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:59,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:59,082 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#C#compaction#572 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:59,085 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#571 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:59,106 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/8facf058f6e447b689a75e6fb32c5287 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8facf058f6e447b689a75e6fb32c5287 2023-04-16 20:55:59,111 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/C of 9853af3382d774c4b02dae6a5c604d40 into 8facf058f6e447b689a75e6fb32c5287(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:59,111 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:59,111 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/C, priority=12, startTime=1681678559059; duration=0sec 2023-04-16 20:55:59,111 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:55:59,111 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:55:59,112 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:55:59,112 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:55:59,112 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:59,112 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1db593803dce49a28a57aad6e6892118, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/62963c595aef48e7b1ff4bc1554f6221, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/30283da273894fea835a386d5c3976e1] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=35.3 K 2023-04-16 20:55:59,113 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 1db593803dce49a28a57aad6e6892118, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681678557580 2023-04-16 20:55:59,113 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 62963c595aef48e7b1ff4bc1554f6221, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681678557657 2023-04-16 20:55:59,113 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 30283da273894fea835a386d5c3976e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681678558278 2023-04-16 20:55:59,120 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#573 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:55:59,135 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/34d8103930b34ab49f50d2046a7cad64 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/34d8103930b34ab49f50d2046a7cad64 2023-04-16 20:55:59,140 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into 34d8103930b34ab49f50d2046a7cad64(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:59,140 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:59,140 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=13, startTime=1681678559059; duration=0sec 2023-04-16 20:55:59,140 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:59,177 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:59,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:59,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:59,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:59,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:59,180 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:59,180 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:59,180 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:59,180 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:59,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,181 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,183 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,183 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:59,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:59,184 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,184 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:59,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:59,185 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:59,185 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:59,185 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-16 20:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:55:59,186 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:59,206 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304161b5be5d1557f4d8fb57299311b10dbe8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161b5be5d1557f4d8fb57299311b10dbe8 2023-04-16 20:55:59,212 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/c9a95d1ea7bc49f4a9e57d6666408314 2023-04-16 20:55:59,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:59,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:59,229 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/175766a40d6044fdbdfb8b9776543013 2023-04-16 20:55:59,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678619234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678619235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678619236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678619237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678619238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678619339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678619339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678619341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678619341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678619341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678619542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678619542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678619543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678619543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678619544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,640 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/ca561e90749d46be9eb9648f089c9319 2023-04-16 20:55:59,644 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/c9a95d1ea7bc49f4a9e57d6666408314 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c9a95d1ea7bc49f4a9e57d6666408314 2023-04-16 20:55:59,647 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c9a95d1ea7bc49f4a9e57d6666408314, entries=150, sequenceid=182, filesize=25.3 K 2023-04-16 20:55:59,648 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/175766a40d6044fdbdfb8b9776543013 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/175766a40d6044fdbdfb8b9776543013 2023-04-16 20:55:59,651 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/175766a40d6044fdbdfb8b9776543013, entries=150, sequenceid=182, filesize=11.8 K 2023-04-16 20:55:59,651 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/ca561e90749d46be9eb9648f089c9319 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ca561e90749d46be9eb9648f089c9319 2023-04-16 20:55:59,654 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ca561e90749d46be9eb9648f089c9319, entries=150, sequenceid=182, filesize=11.8 K 2023-04-16 20:55:59,655 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 9853af3382d774c4b02dae6a5c604d40 in 470ms, sequenceid=182, compaction requested=false 2023-04-16 20:55:59,655 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:59,655 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:59,655 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:55:59,655 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:55:59,655 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,655 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:55:59,655 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:55:59,657 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:59,657 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,657 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,657 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:59,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:55:59,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f3d8481[Count = 0] remaining members to acquire global barrier 2023-04-16 20:55:59,659 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:55:59,659 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,660 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:55:59,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:55:59,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:55:59,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,666 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:55:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:55:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:59,667 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:55:59,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:55:59,669 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:55:59,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:55:59,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:55:59,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:55:59,670 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:55:59,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:55:59,673 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:59,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:55:59,673 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,675 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:59,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,677 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:55:59,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:55:59,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:55:59,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:55:59,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:55:59,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:59,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,677 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:55:59,677 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:55:59,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:55:59,677 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:55:59,777 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:55:59,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:55:59,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:55:59,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:55:59,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:55:59,781 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:55:59,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:59,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:55:59,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:55:59,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,782 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:55:59,782 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:55:59,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,783 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:55:59,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:55:59,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:55:59,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:55:59,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:55:59,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:55:59,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:55:59,784 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:55:59,784 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:55:59,784 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:55:59,785 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 20:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:55:59,785 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:55:59,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:55:59,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:55:59,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678619858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678619861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678619862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678619861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678619862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,907 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/102a79efb4634694a7fe07d970d74df5 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/102a79efb4634694a7fe07d970d74df5 2023-04-16 20:55:59,911 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into 102a79efb4634694a7fe07d970d74df5(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:55:59,911 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:55:59,911 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=13, startTime=1681678559059; duration=0sec 2023-04-16 20:55:59,911 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:55:59,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678619963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678619967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678619967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678619967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:55:59,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:55:59,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678619967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678620165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678620170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678620170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678620170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678620173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,197 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304163550395fa2ce458093693e6f96275ad1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163550395fa2ce458093693e6f96275ad1 2023-04-16 20:56:00,203 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=196, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/c2e53985332c4d2f90055e53ca30407f 2023-04-16 20:56:00,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678620469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678620474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678620474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678620474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678620476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,612 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/1310bd0c441047248f10b140e85c8465 2023-04-16 20:56:00,621 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/cd5ecbc5c4f74c3e9f56b552e18b28a9 2023-04-16 20:56:00,625 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/c2e53985332c4d2f90055e53ca30407f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c2e53985332c4d2f90055e53ca30407f 2023-04-16 20:56:00,628 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c2e53985332c4d2f90055e53ca30407f, entries=150, sequenceid=196, filesize=25.3 K 2023-04-16 20:56:00,629 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/1310bd0c441047248f10b140e85c8465 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1310bd0c441047248f10b140e85c8465 2023-04-16 20:56:00,632 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1310bd0c441047248f10b140e85c8465, entries=150, sequenceid=196, filesize=11.8 K 2023-04-16 20:56:00,633 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/cd5ecbc5c4f74c3e9f56b552e18b28a9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cd5ecbc5c4f74c3e9f56b552e18b28a9 2023-04-16 20:56:00,636 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cd5ecbc5c4f74c3e9f56b552e18b28a9, entries=150, sequenceid=196, filesize=11.8 K 2023-04-16 20:56:00,637 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 9853af3382d774c4b02dae6a5c604d40 in 852ms, sequenceid=196, compaction requested=true 2023-04-16 20:56:00,637 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:00,637 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:00,638 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:00,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:00,639 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c893131[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:00,640 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:00,640 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:00,642 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:00,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:00,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:00,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:00,642 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:00,642 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,643 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:00,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:00,644 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:00,644 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:00,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:00,646 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:00,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:00,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:00,647 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:00,647 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:00,648 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:00,648 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:00,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:00,648 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:00,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:00,649 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:00,650 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:00,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:00,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:00,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:00,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:00,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:00,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:00,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:00,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:00,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:00,652 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:00,652 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:00,652 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:00,653 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:00,653 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:00,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:00,753 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:00,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:00,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:00,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:00,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:00,756 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:00,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:00,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:00,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:00,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:00,757 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:00,759 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:00,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:00,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:00,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:00,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:00,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:00,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:00,760 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:00,760 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:00,761 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-16 20:56:00,761 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:00,761 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:00,761 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:00,761 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:00,761 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:00,761 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:00,788 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416560324370660428f884c98b1c7a12def to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416560324370660428f884c98b1c7a12def 2023-04-16 20:56:00,793 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=216, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/ddb9ccf568564db2abbd4943da7c8324 2023-04-16 20:56:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:00,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:00,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678620982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678620983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678620984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678620984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:00,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:00,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678620985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678621086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678621086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678621088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678621089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678621088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,202 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/6dbf9453875943c58b7545c200690325 2023-04-16 20:56:01,217 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/ceebbbec6a95483cb8642f6d066e7ce3 2023-04-16 20:56:01,221 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/ddb9ccf568564db2abbd4943da7c8324 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ddb9ccf568564db2abbd4943da7c8324 2023-04-16 20:56:01,224 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ddb9ccf568564db2abbd4943da7c8324, entries=150, sequenceid=216, filesize=25.3 K 2023-04-16 20:56:01,225 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/6dbf9453875943c58b7545c200690325 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/6dbf9453875943c58b7545c200690325 2023-04-16 20:56:01,228 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/6dbf9453875943c58b7545c200690325, entries=150, sequenceid=216, filesize=11.8 K 2023-04-16 20:56:01,229 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/ceebbbec6a95483cb8642f6d066e7ce3 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ceebbbec6a95483cb8642f6d066e7ce3 2023-04-16 20:56:01,232 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ceebbbec6a95483cb8642f6d066e7ce3, entries=150, sequenceid=216, filesize=11.8 K 2023-04-16 20:56:01,233 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 9853af3382d774c4b02dae6a5c604d40 in 472ms, sequenceid=216, compaction requested=true 2023-04-16 20:56:01,233 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:01,233 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:01,235 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,235 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:01,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad25f99[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:01,237 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:01,237 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,243 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:01,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,244 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:01,246 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,247 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:01,250 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:01,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:01,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:01,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:01,250 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:01,251 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,251 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,251 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,254 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:01,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:01,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:01,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,257 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:01,257 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:01,256 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:01,291 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 20:56:01,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:01,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:01,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:01,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:01,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:01,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:01,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678621304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678621305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678621306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678621307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678621308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,357 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:01,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:01,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:01,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:01,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:01,361 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:01,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:01,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:01,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,363 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,363 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:01,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:01,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:01,364 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:01,364 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,364 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:01,364 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:01,364 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:01,365 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,365 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:01,365 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:01,365 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:01,367 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:01,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77bf3078[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:01,369 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:01,369 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,371 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,371 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:01,371 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:01,371 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:01,371 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,371 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:01,372 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:01,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:01,374 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:01,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:01,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:01,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:01,374 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:01,376 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,376 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,376 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,377 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,378 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,381 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:01,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:01,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,381 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:01,381 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:01,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,381 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:01,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678621409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678621410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678621410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678621412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678621413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,481 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:01,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:01,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:01,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:01,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:01,485 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:01,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:01,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:01,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,485 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,487 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:01,489 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:01,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:01,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:01,491 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,491 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:01,491 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:01,491 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:01,492 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,492 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:01,492 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:01,493 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,493 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:01,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4495ab82[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:01,495 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:01,495 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,496 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:01,497 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,497 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:01,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:01,499 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,499 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:01,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:01,502 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:01,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:01,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:01,502 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:01,502 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:01,504 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,504 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,504 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,504 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,506 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:01,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:01,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,508 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,508 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:01,508 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:01,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,608 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:01,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:01,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:01,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:01,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:01,612 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:01,612 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,612 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:01,612 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:01,612 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,612 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,614 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,614 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,614 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,614 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678621612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:01,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:01,616 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,616 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:01,616 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:01,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,616 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678621614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:01,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678621614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:01,617 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:01,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678621615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,619 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678621618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:01,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41901a98[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:01,621 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:01,622 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,623 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:01,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:01,623 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,623 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:01,623 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:01,625 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:01,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:01,628 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:01,628 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:01,628 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:01,628 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:01,628 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:01,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,632 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,633 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,634 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,637 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,637 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:01,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:01,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:01,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,638 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:01,638 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:01,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,704 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041693d95b269f07424abea9bb53ebe60298 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041693d95b269f07424abea9bb53ebe60298 2023-04-16 20:56:01,709 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/b630a135cfe44963b749c0ad452a2280 2023-04-16 20:56:01,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/9b5572e2907f49e39a2907891d808b18 2023-04-16 20:56:01,738 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:01,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:01,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:01,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:01,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:01,753 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:01,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:01,753 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:01,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,754 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,755 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,755 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,756 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:01,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:01,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:01,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:01,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:01,772 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:01,772 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:01,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/e94da69735344add89381974d11eba68 2023-04-16 20:56:01,772 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,772 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:01,772 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:01,772 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:01,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:01,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:01,773 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:01,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,775 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,775 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/b630a135cfe44963b749c0ad452a2280 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b630a135cfe44963b749c0ad452a2280 2023-04-16 20:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3946aba4[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:01,779 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:01,779 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,780 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:01,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,781 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:01,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b630a135cfe44963b749c0ad452a2280, entries=200, sequenceid=231, filesize=32.1 K 2023-04-16 20:56:01,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/9b5572e2907f49e39a2907891d808b18 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/9b5572e2907f49e39a2907891d808b18 2023-04-16 20:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,783 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/9b5572e2907f49e39a2907891d808b18, entries=150, sequenceid=231, filesize=11.8 K 2023-04-16 20:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/e94da69735344add89381974d11eba68 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e94da69735344add89381974d11eba68 2023-04-16 20:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:01,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:01,786 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:01,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:01,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:01,786 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:01,786 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:01,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:01,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,788 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,789 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e94da69735344add89381974d11eba68, entries=150, sequenceid=231, filesize=11.8 K 2023-04-16 20:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,789 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:01,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9853af3382d774c4b02dae6a5c604d40 in 499ms, sequenceid=231, compaction requested=true 2023-04-16 20:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:01,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:01,790 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:56:01,790 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:01,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:01,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,792 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136778 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:56:01,792 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:56:01,792 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60488 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:56:01,792 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,792 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:56:01,792 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/102a79efb4634694a7fe07d970d74df5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c9a95d1ea7bc49f4a9e57d6666408314, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c2e53985332c4d2f90055e53ca30407f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ddb9ccf568564db2abbd4943da7c8324, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b630a135cfe44963b749c0ad452a2280] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=133.6 K 2023-04-16 20:56:01,792 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,792 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/34d8103930b34ab49f50d2046a7cad64, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/175766a40d6044fdbdfb8b9776543013, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1310bd0c441047248f10b140e85c8465, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/6dbf9453875943c58b7545c200690325, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/9b5572e2907f49e39a2907891d808b18] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=59.1 K 2023-04-16 20:56:01,792 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 34d8103930b34ab49f50d2046a7cad64, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681678558278 2023-04-16 20:56:01,793 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 175766a40d6044fdbdfb8b9776543013, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681678558609 2023-04-16 20:56:01,793 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 1310bd0c441047248f10b140e85c8465, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681678559233 2023-04-16 20:56:01,793 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 102a79efb4634694a7fe07d970d74df5, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681678558278 2023-04-16 20:56:01,794 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c9a95d1ea7bc49f4a9e57d6666408314, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681678558609 2023-04-16 20:56:01,794 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6dbf9453875943c58b7545c200690325, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681678559860 2023-04-16 20:56:01,794 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c2e53985332c4d2f90055e53ca30407f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681678559233 2023-04-16 20:56:01,794 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ddb9ccf568564db2abbd4943da7c8324, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681678559860 2023-04-16 20:56:01,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,795 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:01,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:01,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:01,795 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting b630a135cfe44963b749c0ad452a2280, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681678560980 2023-04-16 20:56:01,795 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:01,795 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:01,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:01,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:01,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,795 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 9b5572e2907f49e39a2907891d808b18, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681678560983 2023-04-16 20:56:01,795 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:01,807 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#587 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:01,813 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#586 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:01,895 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:01,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:01,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:01,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:01,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:01,898 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:01,898 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,898 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:01,898 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:01,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,899 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,900 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:01,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:01,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,900 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:01,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:01,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:01,902 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:01,902 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:01,902 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-16 20:56:01,902 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:01,902 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:01,902 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:01,902 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:01,902 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:01,902 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:01,918 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416c5f3dc5fa0394a61921b79a2ab23787b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c5f3dc5fa0394a61921b79a2ab23787b 2023-04-16 20:56:01,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:01,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:01,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678621929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678621930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678621931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678621932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:01,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:01,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678621933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678622034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678622034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678622034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678622035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678622037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,232 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/f6d75490a3a44a14b3e295bb73aad011 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f6d75490a3a44a14b3e295bb73aad011 2023-04-16 20:56:02,237 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into f6d75490a3a44a14b3e295bb73aad011(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:02,237 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:02,237 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=11, startTime=1681678561790; duration=0sec 2023-04-16 20:56:02,237 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:02,237 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:56:02,239 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:56:02,239 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/C is initiating minor compaction (all files) 2023-04-16 20:56:02,239 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/C in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:02,239 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8facf058f6e447b689a75e6fb32c5287, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ca561e90749d46be9eb9648f089c9319, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cd5ecbc5c4f74c3e9f56b552e18b28a9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ceebbbec6a95483cb8642f6d066e7ce3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e94da69735344add89381974d11eba68] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=59.0 K 2023-04-16 20:56:02,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678622237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678622237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678622237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,240 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8facf058f6e447b689a75e6fb32c5287, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681678558278 2023-04-16 20:56:02,241 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ca561e90749d46be9eb9648f089c9319, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681678558609 2023-04-16 20:56:02,241 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting cd5ecbc5c4f74c3e9f56b552e18b28a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681678559233 2023-04-16 20:56:02,242 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ceebbbec6a95483cb8642f6d066e7ce3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681678559860 2023-04-16 20:56:02,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678622238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,242 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e94da69735344add89381974d11eba68, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681678560983 2023-04-16 20:56:02,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678622240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,244 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/a65af645b2d54fcf83027ebfed929062 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/a65af645b2d54fcf83027ebfed929062 2023-04-16 20:56:02,249 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into a65af645b2d54fcf83027ebfed929062(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:02,249 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:02,249 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=11, startTime=1681678561790; duration=0sec 2023-04-16 20:56:02,249 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:02,250 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#C#compaction#589 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:02,323 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=248, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/86ee031a103745c38fa6d7471a639bd9 2023-04-16 20:56:02,333 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/b6756ca5e4764e938e2c3f6a9fe46ffd 2023-04-16 20:56:02,341 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/952f6265721049ffa8b5f7affdc85e5f 2023-04-16 20:56:02,345 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/86ee031a103745c38fa6d7471a639bd9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/86ee031a103745c38fa6d7471a639bd9 2023-04-16 20:56:02,348 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/86ee031a103745c38fa6d7471a639bd9, entries=150, sequenceid=248, filesize=25.3 K 2023-04-16 20:56:02,349 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/b6756ca5e4764e938e2c3f6a9fe46ffd as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b6756ca5e4764e938e2c3f6a9fe46ffd 2023-04-16 20:56:02,352 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b6756ca5e4764e938e2c3f6a9fe46ffd, entries=150, sequenceid=248, filesize=11.8 K 2023-04-16 20:56:02,353 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/952f6265721049ffa8b5f7affdc85e5f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/952f6265721049ffa8b5f7affdc85e5f 2023-04-16 20:56:02,355 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/952f6265721049ffa8b5f7affdc85e5f, entries=150, sequenceid=248, filesize=11.8 K 2023-04-16 20:56:02,356 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9853af3382d774c4b02dae6a5c604d40 in 454ms, sequenceid=248, compaction requested=false 2023-04-16 20:56:02,356 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:02,356 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:02,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:02,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:02,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:02,356 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,359 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:02,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:02,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f47004[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:02,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:02,361 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,362 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:02,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:02,362 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:02,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,362 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:02,363 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:02,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:02,366 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:02,366 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:02,366 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:02,366 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:02,366 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:02,368 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,368 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:02,368 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,368 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:02,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,370 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,372 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:02,372 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:02,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:02,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:02,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:02,372 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:02,373 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:02,372 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:02,372 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:02,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:02,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,373 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:02,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:02,473 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:02,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:02,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:02,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:02,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:02,477 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:02,477 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:02,478 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:02,478 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:02,478 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,478 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:02,479 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:02,479 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:02,480 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:02,480 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:02,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:02,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:02,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:02,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:02,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:02,482 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:02,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:02,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:02,482 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-16 20:56:02,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:02,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:02,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:02,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:02,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:02,482 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:02,499 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416acfff2d5e5f147a6b00339b31a4bb4c5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416acfff2d5e5f147a6b00339b31a4bb4c5 2023-04-16 20:56:02,505 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=265, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/c96838fe89eb40d98abdf27cf3aa4193 2023-04-16 20:56:02,517 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/d97a4c03611d4e7fadd102bcd7c9e532 2023-04-16 20:56:02,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:02,543 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:02,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678622552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678622553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678622556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678622556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678622556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,659 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/ac25876d45684ca5b7d462a7310d62d0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ac25876d45684ca5b7d462a7310d62d0 2023-04-16 20:56:02,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678622658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678622658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678622662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678622662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678622662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,666 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/C of 9853af3382d774c4b02dae6a5c604d40 into ac25876d45684ca5b7d462a7310d62d0(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:02,666 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:02,666 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/C, priority=11, startTime=1681678561790; duration=0sec 2023-04-16 20:56:02,666 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:02,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678622862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678622862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678622864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678622865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:02,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678622866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,926 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/536904cc5a684e12b23512fd9ced6cae 2023-04-16 20:56:02,930 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/c96838fe89eb40d98abdf27cf3aa4193 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c96838fe89eb40d98abdf27cf3aa4193 2023-04-16 20:56:02,933 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c96838fe89eb40d98abdf27cf3aa4193, entries=150, sequenceid=265, filesize=25.4 K 2023-04-16 20:56:02,934 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/d97a4c03611d4e7fadd102bcd7c9e532 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d97a4c03611d4e7fadd102bcd7c9e532 2023-04-16 20:56:02,937 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d97a4c03611d4e7fadd102bcd7c9e532, entries=150, sequenceid=265, filesize=11.9 K 2023-04-16 20:56:02,938 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/536904cc5a684e12b23512fd9ced6cae as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/536904cc5a684e12b23512fd9ced6cae 2023-04-16 20:56:02,941 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/536904cc5a684e12b23512fd9ced6cae, entries=150, sequenceid=265, filesize=11.9 K 2023-04-16 20:56:02,942 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9853af3382d774c4b02dae6a5c604d40 in 459ms, sequenceid=265, compaction requested=true 2023-04-16 20:56:02,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:02,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:02,943 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed404c3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:02,945 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:02,945 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,947 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:02,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:02,947 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:02,947 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,947 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:02,949 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:02,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:02,953 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:02,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:02,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:02,953 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:02,953 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:02,954 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,954 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:02,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,954 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:02,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:02,955 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,957 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:02,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:02,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:02,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:02,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:02,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:02,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:02,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:02,962 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:02,962 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:02,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:02,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:02,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:02,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:02,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,062 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:03,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:03,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:03,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:03,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:03,066 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:03,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:03,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:03,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,066 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,072 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,072 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,072 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,072 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:03,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:03,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:03,074 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,074 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:03,074 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-16 20:56:03,074 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:03,074 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:03,074 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:03,074 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:03,075 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:03,075 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:03,088 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304166ac830aa52c7491b9cc103600d0e5096 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166ac830aa52c7491b9cc103600d0e5096 2023-04-16 20:56:03,093 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=283, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/3f511669c2af4ea0b3f5c03dcfb7640b 2023-04-16 20:56:03,142 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/70d8851e511d40e4be5dcc77f1f9a8c4 2023-04-16 20:56:03,154 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/d22bcb8c03484fa48db84e497127d443 2023-04-16 20:56:03,159 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/3f511669c2af4ea0b3f5c03dcfb7640b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3f511669c2af4ea0b3f5c03dcfb7640b 2023-04-16 20:56:03,163 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3f511669c2af4ea0b3f5c03dcfb7640b, entries=150, sequenceid=283, filesize=25.4 K 2023-04-16 20:56:03,163 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/70d8851e511d40e4be5dcc77f1f9a8c4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/70d8851e511d40e4be5dcc77f1f9a8c4 2023-04-16 20:56:03,166 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/70d8851e511d40e4be5dcc77f1f9a8c4, entries=150, sequenceid=283, filesize=11.9 K 2023-04-16 20:56:03,167 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/d22bcb8c03484fa48db84e497127d443 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d22bcb8c03484fa48db84e497127d443 2023-04-16 20:56:03,173 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d22bcb8c03484fa48db84e497127d443, entries=150, sequenceid=283, filesize=11.9 K 2023-04-16 20:56:03,173 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 9853af3382d774c4b02dae6a5c604d40 in 99ms, sequenceid=283, compaction requested=true 2023-04-16 20:56:03,173 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:03,173 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,174 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:03,174 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:03,174 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,174 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:03,174 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:03,175 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,175 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,176 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f51815b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:03,177 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:03,177 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,179 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,179 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:03,179 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:03,179 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:03,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,179 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:03,180 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,180 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:03,180 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,180 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:03,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:03,183 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:03,183 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:03,183 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:03,183 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:03,183 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:03,188 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,188 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,189 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,189 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,191 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,193 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:03,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:03,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,193 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:03,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,193 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,194 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,193 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:03,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:03,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:03,199 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:56:03,200 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:03,200 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:03,200 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:03,200 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:03,200 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:03,200 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:03,216 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041618a2e6912c7a49df989ec6860936c962 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041618a2e6912c7a49df989ec6860936c962 2023-04-16 20:56:03,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678623218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678623218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678563216, max timestamp=1681678563222Num uniques 859; 2023-04-16 20:56:03,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678563216, max timestamp=1681678563222Num uniques 859; 2023-04-16 20:56:03,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:03,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678563216, max timestamp=1681678563222Num uniques 859; 2023-04-16 20:56:03,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:03,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:03,229 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=294, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/c21b67d588d0416cb8e422fe13d8889e 2023-04-16 20:56:03,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/0c0e06f0d1694a4ba2c8ee56dcc19d71 2023-04-16 20:56:03,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678623244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678623246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678623248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,294 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:03,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:03,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:03,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:03,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:03,297 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:03,297 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,297 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:03,298 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:03,298 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,298 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,299 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,299 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,300 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,300 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:03,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:03,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:03,301 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,301 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:03,301 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:03,301 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:03,302 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,302 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:03,302 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:03,303 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,303 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,303 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,303 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:03,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:03,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c6129e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:03,310 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:03,310 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,312 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:03,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:03,312 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:03,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,312 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:03,314 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:03,314 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,314 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,314 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:03,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:03,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:03,317 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:03,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:03,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:03,317 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:03,317 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:03,318 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,318 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,318 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,319 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,321 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,324 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:03,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:03,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,325 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:03,325 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:03,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:03,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,324 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,325 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,325 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,325 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,325 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,325 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678623325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678623325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678623349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678623350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678623353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,425 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:03,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:03,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:03,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:03,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:03,428 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:03,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:03,428 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:03,429 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,429 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,431 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,431 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,432 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,432 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:03,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:03,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:03,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:03,433 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,433 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:03,433 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:03,433 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:03,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:03,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:03,434 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:03,435 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,436 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,436 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,436 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:03,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72624b2[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:03,438 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:03,438 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,439 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:03,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,440 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:03,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:03,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,441 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:03,441 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:03,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:03,445 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:03,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:03,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:03,445 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:03,445 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:03,448 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,448 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,449 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,450 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,451 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:03,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:03,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,458 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:03,458 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:03,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:03,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40754 deadline: 1681678623530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40750 deadline: 1681678623531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40748 deadline: 1681678623553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40762 deadline: 1681678623554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,558 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:03,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:03,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:03,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:03,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:03,561 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:03,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:03,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:03,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40784 deadline: 1681678623557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,563 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,563 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,563 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,563 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:03,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:03,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:03,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:03,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:03,568 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,568 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:03,568 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:03,568 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:03,574 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,574 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:03,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79aaca89[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:03,578 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:03,578 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,579 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:03,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:03,579 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:03,579 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,579 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:03,581 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,581 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:03,581 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,581 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:03,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:03,584 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:03,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:03,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:03,584 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:03,584 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:03,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,586 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,587 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,588 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,591 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:03,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:03,592 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,592 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:03,592 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:03,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:03,651 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/e3e27a0af1734605b96323327532eec2 2023-04-16 20:56:03,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/c21b67d588d0416cb8e422fe13d8889e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c21b67d588d0416cb8e422fe13d8889e 2023-04-16 20:56:03,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c21b67d588d0416cb8e422fe13d8889e, entries=400, sequenceid=294, filesize=59.8 K 2023-04-16 20:56:03,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/0c0e06f0d1694a4ba2c8ee56dcc19d71 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0c0e06f0d1694a4ba2c8ee56dcc19d71 2023-04-16 20:56:03,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0c0e06f0d1694a4ba2c8ee56dcc19d71, entries=150, sequenceid=294, filesize=11.9 K 2023-04-16 20:56:03,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/e3e27a0af1734605b96323327532eec2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e3e27a0af1734605b96323327532eec2 2023-04-16 20:56:03,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e3e27a0af1734605b96323327532eec2, entries=150, sequenceid=294, filesize=11.9 K 2023-04-16 20:56:03,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9853af3382d774c4b02dae6a5c604d40 in 469ms, sequenceid=294, compaction requested=true 2023-04-16 20:56:03,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:03,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:03,669 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:56:03,669 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:56:03,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:03,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:03,670 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 165566 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:56:03,670 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:56:03,670 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,670 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/a65af645b2d54fcf83027ebfed929062, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/86ee031a103745c38fa6d7471a639bd9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c96838fe89eb40d98abdf27cf3aa4193, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3f511669c2af4ea0b3f5c03dcfb7640b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c21b67d588d0416cb8e422fe13d8889e] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=161.7 K 2023-04-16 20:56:03,671 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61058 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:56:03,671 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:56:03,671 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,671 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f6d75490a3a44a14b3e295bb73aad011, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b6756ca5e4764e938e2c3f6a9fe46ffd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d97a4c03611d4e7fadd102bcd7c9e532, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/70d8851e511d40e4be5dcc77f1f9a8c4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0c0e06f0d1694a4ba2c8ee56dcc19d71] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=59.6 K 2023-04-16 20:56:03,671 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting a65af645b2d54fcf83027ebfed929062, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681678560983 2023-04-16 20:56:03,672 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting f6d75490a3a44a14b3e295bb73aad011, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681678560983 2023-04-16 20:56:03,672 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 86ee031a103745c38fa6d7471a639bd9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681678561301 2023-04-16 20:56:03,672 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b6756ca5e4764e938e2c3f6a9fe46ffd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681678561301 2023-04-16 20:56:03,673 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c96838fe89eb40d98abdf27cf3aa4193, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678561926 2023-04-16 20:56:03,673 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d97a4c03611d4e7fadd102bcd7c9e532, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678561926 2023-04-16 20:56:03,673 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3f511669c2af4ea0b3f5c03dcfb7640b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681678562550 2023-04-16 20:56:03,674 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 70d8851e511d40e4be5dcc77f1f9a8c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681678562550 2023-04-16 20:56:03,674 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0c0e06f0d1694a4ba2c8ee56dcc19d71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681678563194 2023-04-16 20:56:03,674 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c21b67d588d0416cb8e422fe13d8889e, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681678563190 2023-04-16 20:56:03,687 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#602 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:03,692 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:03,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:03,695 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#601 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:03,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:03,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:03,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:03,708 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:03,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:03,708 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:03,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,709 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,710 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,710 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,711 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:03,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:03,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:03,716 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:03,716 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,716 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:03,720 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:03,720 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:03,720 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,720 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:03,720 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 20:56:03,720 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:03,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:03,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:03,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:03,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:03,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:03,773 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304169a861cfc58564cfc8e16b2a8edd494d4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169a861cfc58564cfc8e16b2a8edd494d4 2023-04-16 20:56:03,779 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=330, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/7ca748abe6064bf7af8f48a688151b71 2023-04-16 20:56:03,794 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/ade1a505568048e4bc14944b1c11f6a0 2023-04-16 20:56:03,805 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/10917cc419b34a3d9d1c108152f20908 2023-04-16 20:56:03,809 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/7ca748abe6064bf7af8f48a688151b71 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7ca748abe6064bf7af8f48a688151b71 2023-04-16 20:56:03,814 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7ca748abe6064bf7af8f48a688151b71, entries=150, sequenceid=330, filesize=25.4 K 2023-04-16 20:56:03,815 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/ade1a505568048e4bc14944b1c11f6a0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ade1a505568048e4bc14944b1c11f6a0 2023-04-16 20:56:03,819 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ade1a505568048e4bc14944b1c11f6a0, entries=150, sequenceid=330, filesize=11.9 K 2023-04-16 20:56:03,819 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/10917cc419b34a3d9d1c108152f20908 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/10917cc419b34a3d9d1c108152f20908 2023-04-16 20:56:03,823 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/10917cc419b34a3d9d1c108152f20908, entries=150, sequenceid=330, filesize=11.9 K 2023-04-16 20:56:03,824 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 9853af3382d774c4b02dae6a5c604d40 in 104ms, sequenceid=330, compaction requested=true 2023-04-16 20:56:03,824 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:03,824 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:03,826 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,827 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:03,829 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:03,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a62c7be[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:03,832 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:03,833 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,834 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,834 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:03,834 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:03,834 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:03,834 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,834 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:03,835 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:03,835 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,835 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:03,835 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:03,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:03,838 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:03,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:03,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:03,838 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:03,838 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:03,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,840 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,841 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,842 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,845 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:03,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:03,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,846 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,846 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:03,846 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:03,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:03,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:03,869 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:56:03,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:03,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:03,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:03,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:03,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:03,870 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:03,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678563888, max timestamp=1681678563899Num uniques 859; 2023-04-16 20:56:03,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678563888, max timestamp=1681678563899Num uniques 859; 2023-04-16 20:56:03,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:03,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678563888, max timestamp=1681678563899Num uniques 859; 2023-04-16 20:56:03,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:03,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:03,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678623918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678623918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678623918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678623919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:03,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678623923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,946 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:03,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:03,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:03,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:03,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:03,949 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:03,949 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,949 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:03,949 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:03,949 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,949 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,951 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,951 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:03,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:03,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:03,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:03,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:03,952 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:03,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:03,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:03,956 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:03,958 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:03,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49936a8e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:03,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:03,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,962 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:03,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:03,962 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:03,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,962 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:03,963 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:03,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,963 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:03,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:03,966 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:03,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:03,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:03,966 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:03,966 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:03,968 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:03,968 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,968 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:03,968 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:03,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:03,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,970 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:03,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:03,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:03,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:03,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:03,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:03,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:03,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:03,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:03,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:03,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:03,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:03,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:03,972 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:03,973 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:03,973 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:03,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:03,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:04,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678624025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678624025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678624025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678624025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678624027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,073 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:04,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:04,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:04,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:04,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:04,076 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:04,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:04,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:04,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:04,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,076 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,078 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:04,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:04,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,079 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:04,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:04,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:04,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:04,080 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:04,080 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:04,080 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:04,080 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:04,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:04,081 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,081 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:04,081 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:04,081 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:04,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:04,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:04,081 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:04,085 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:04,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fa6d4ba[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:04,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:04,087 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,088 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,088 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,088 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:04,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:04,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:04,088 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:04,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:04,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:04,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:04,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:04,093 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:04,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:04,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:04,093 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:04,093 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:04,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,094 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:04,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:04,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:04,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,095 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,096 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,098 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,098 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,098 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:04,098 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:04,098 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:04,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:04,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:04,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:04,098 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:04,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,099 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,099 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:04,099 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:04,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:04,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:04,105 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/434850c28ff046d4b2c025560ab8261f as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/434850c28ff046d4b2c025560ab8261f 2023-04-16 20:56:04,109 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into 434850c28ff046d4b2c025560ab8261f(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:04,109 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:04,109 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=11, startTime=1681678563669; duration=0sec 2023-04-16 20:56:04,109 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:04,110 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:56:04,111 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73210 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:56:04,111 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/C is initiating minor compaction (all files) 2023-04-16 20:56:04,111 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/C in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,111 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ac25876d45684ca5b7d462a7310d62d0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/952f6265721049ffa8b5f7affdc85e5f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/536904cc5a684e12b23512fd9ced6cae, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d22bcb8c03484fa48db84e497127d443, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e3e27a0af1734605b96323327532eec2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/10917cc419b34a3d9d1c108152f20908] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=71.5 K 2023-04-16 20:56:04,111 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ac25876d45684ca5b7d462a7310d62d0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681678560983 2023-04-16 20:56:04,112 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 952f6265721049ffa8b5f7affdc85e5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681678561301 2023-04-16 20:56:04,112 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 536904cc5a684e12b23512fd9ced6cae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681678561926 2023-04-16 20:56:04,112 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d22bcb8c03484fa48db84e497127d443, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1681678562550 2023-04-16 20:56:04,113 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e3e27a0af1734605b96323327532eec2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681678563194 2023-04-16 20:56:04,113 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 10917cc419b34a3d9d1c108152f20908, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681678563240 2023-04-16 20:56:04,120 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#C#compaction#607 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:04,165 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/95da552ee04e415b9604801ca84d9ed6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/95da552ee04e415b9604801ca84d9ed6 2023-04-16 20:56:04,169 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into 95da552ee04e415b9604801ca84d9ed6(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:04,169 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:04,169 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=11, startTime=1681678563668; duration=0sec 2023-04-16 20:56:04,169 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:04,199 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:04,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:04,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:04,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:04,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:04,202 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:04,202 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:04,202 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:04,202 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:04,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,203 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,204 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:04,204 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:04,205 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,205 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:04,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:04,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:04,206 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,206 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:04,206 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:04,206 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:04,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:04,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:04,207 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:04,209 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,210 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:04,210 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:04,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5077a745[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:04,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:04,212 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,213 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:04,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:04,213 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:04,213 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,213 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:04,214 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,214 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:04,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:04,215 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:04,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:04,217 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:04,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:04,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:04,217 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:04,217 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:04,218 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,218 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,218 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:04,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,220 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,221 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:04,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:04,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:04,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:04,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:04,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:04,224 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:04,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,224 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,224 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:04,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:04,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:04,224 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:04,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678624229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678624229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678624230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678624230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678624230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,309 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416fae5671b1526459c9dfd12f39d582735 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416fae5671b1526459c9dfd12f39d582735 2023-04-16 20:56:04,325 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:04,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:04,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:04,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:04,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:04,328 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:04,328 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:04,328 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:04,328 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:04,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,329 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=340, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/5b6a5a3dcda547d6b56feee69072111e 2023-04-16 20:56:04,329 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,331 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:04,331 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:04,331 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,331 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:04,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:04,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:04,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:04,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:04,333 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,333 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:04,333 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:04,333 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:04,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:04,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:04,333 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:04,335 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,335 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,336 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,336 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:04,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd4c5c2[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:04,338 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:04,338 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/b67b0936a8cf49f5b34fd2a9eb5c00e1 2023-04-16 20:56:04,339 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:04,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:04,339 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:04,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,339 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:04,341 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,341 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,341 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:04,342 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:04,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:04,344 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:04,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:04,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:04,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:04,345 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:04,346 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,346 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:04,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,347 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,348 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/cf901dca34d8436590a5f6db99ed3302 2023-04-16 20:56:04,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,351 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,351 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,351 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:04,352 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:04,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:04,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:04,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:04,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:04,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:04,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,352 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:04,353 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:04,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/5b6a5a3dcda547d6b56feee69072111e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/5b6a5a3dcda547d6b56feee69072111e 2023-04-16 20:56:04,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,353 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,353 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:04,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/5b6a5a3dcda547d6b56feee69072111e, entries=200, sequenceid=340, filesize=32.3 K 2023-04-16 20:56:04,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/b67b0936a8cf49f5b34fd2a9eb5c00e1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b67b0936a8cf49f5b34fd2a9eb5c00e1 2023-04-16 20:56:04,359 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b67b0936a8cf49f5b34fd2a9eb5c00e1, entries=150, sequenceid=340, filesize=11.9 K 2023-04-16 20:56:04,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/cf901dca34d8436590a5f6db99ed3302 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cf901dca34d8436590a5f6db99ed3302 2023-04-16 20:56:04,363 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cf901dca34d8436590a5f6db99ed3302, entries=150, sequenceid=340, filesize=11.9 K 2023-04-16 20:56:04,363 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9853af3382d774c4b02dae6a5c604d40 in 494ms, sequenceid=340, compaction requested=true 2023-04-16 20:56:04,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:04,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:04,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:04,364 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:56:04,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:56:04,364 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:56:04,364 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:56:04,365 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,365 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/95da552ee04e415b9604801ca84d9ed6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7ca748abe6064bf7af8f48a688151b71, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/5b6a5a3dcda547d6b56feee69072111e] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=83.8 K 2023-04-16 20:56:04,365 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 95da552ee04e415b9604801ca84d9ed6, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681678563194 2023-04-16 20:56:04,365 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7ca748abe6064bf7af8f48a688151b71, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681678563240 2023-04-16 20:56:04,365 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 5b6a5a3dcda547d6b56feee69072111e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681678563867 2023-04-16 20:56:04,372 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#610 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:04,453 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:04,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:04,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:04,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:04,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:04,456 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:04,456 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:04,456 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:04,456 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:04,456 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,456 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,457 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:04,457 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:04,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,458 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:04,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:04,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:04,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:04,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:04,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:04,459 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:04,459 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,459 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:04,459 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-16 20:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:04,528 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/51003de5977e4465ad1330b95d15070c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/51003de5977e4465ad1330b95d15070c 2023-04-16 20:56:04,532 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/C of 9853af3382d774c4b02dae6a5c604d40 into 51003de5977e4465ad1330b95d15070c(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:04,532 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:04,532 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/C, priority=10, startTime=1681678563669; duration=0sec 2023-04-16 20:56:04,532 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:56:04,532 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:56:04,533 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:56:04,533 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:56:04,533 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,533 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/434850c28ff046d4b2c025560ab8261f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ade1a505568048e4bc14944b1c11f6a0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b67b0936a8cf49f5b34fd2a9eb5c00e1] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=36.3 K 2023-04-16 20:56:04,534 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 434850c28ff046d4b2c025560ab8261f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681678563194 2023-04-16 20:56:04,534 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ade1a505568048e4bc14944b1c11f6a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681678563240 2023-04-16 20:56:04,535 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b67b0936a8cf49f5b34fd2a9eb5c00e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681678563867 2023-04-16 20:56:04,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:04,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:04,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678624540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678624541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678624544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,547 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#612 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:04,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678624545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678624545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,554 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/43ed3dc847ef440b8369dbc1aaa819fb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/43ed3dc847ef440b8369dbc1aaa819fb 2023-04-16 20:56:04,557 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into 43ed3dc847ef440b8369dbc1aaa819fb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:04,558 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:04,558 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=13, startTime=1681678564364; duration=0sec 2023-04-16 20:56:04,558 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:04,558 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-16 20:56:04,558 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-16 20:56:04,558 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-16 20:56:04,559 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. because compaction request was cancelled 2023-04-16 20:56:04,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678624646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678624647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678624647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678624650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678624650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,790 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/2b92c60bd0494472b257320163258119 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2b92c60bd0494472b257320163258119 2023-04-16 20:56:04,795 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into 2b92c60bd0494472b257320163258119(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:04,795 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:04,795 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=13, startTime=1681678564364; duration=0sec 2023-04-16 20:56:04,795 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:04,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678624850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678624852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678624853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678624853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:04,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678624858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,873 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416f8fdee967ab94e51a4080b811a781a83 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f8fdee967ab94e51a4080b811a781a83 2023-04-16 20:56:04,888 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=378, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/60537b2702924acbacccf9eafd6c5e63 2023-04-16 20:56:04,918 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/5bd8c2bb01c141548250b80f34ec9420 2023-04-16 20:56:04,933 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/206e14bf06a44445864681e3736b3070 2023-04-16 20:56:04,937 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/60537b2702924acbacccf9eafd6c5e63 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/60537b2702924acbacccf9eafd6c5e63 2023-04-16 20:56:04,941 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/60537b2702924acbacccf9eafd6c5e63, entries=150, sequenceid=378, filesize=25.4 K 2023-04-16 20:56:04,942 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/5bd8c2bb01c141548250b80f34ec9420 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5bd8c2bb01c141548250b80f34ec9420 2023-04-16 20:56:04,945 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5bd8c2bb01c141548250b80f34ec9420, entries=150, sequenceid=378, filesize=11.9 K 2023-04-16 20:56:04,946 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/206e14bf06a44445864681e3736b3070 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/206e14bf06a44445864681e3736b3070 2023-04-16 20:56:04,950 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/206e14bf06a44445864681e3736b3070, entries=150, sequenceid=378, filesize=11.9 K 2023-04-16 20:56:04,951 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 9853af3382d774c4b02dae6a5c604d40 in 492ms, sequenceid=378, compaction requested=true 2023-04-16 20:56:04,951 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:04,951 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:04,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:04,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:04,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:04,951 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,957 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,958 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec10dcc[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:04,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:04,960 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,964 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:04,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:04,964 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:04,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,964 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:04,966 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:04,966 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,966 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,966 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:04,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:04,969 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:04,969 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:04,969 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:04,969 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:04,969 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:04,970 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:04,971 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,971 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,971 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:04,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:04,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,972 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:04,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:04,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,975 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:04,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:04,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:04,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:04,976 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:04,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:04,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:04,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:04,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:04,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:04,976 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:04,976 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:04,976 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:04,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:05,076 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:05,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:05,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:05,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:05,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:05,080 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:05,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:05,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:05,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:05,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,080 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,081 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,081 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:05,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:05,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:05,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,082 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:05,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:05,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:05,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:05,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:05,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:05,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:05,083 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:05,084 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:05,095 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416096907b68e55455cabbca11c7ef1dd50 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416096907b68e55455cabbca11c7ef1dd50 2023-04-16 20:56:05,101 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=391, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/eb8013ea1e5a40bd8b04158ddd59af2e 2023-04-16 20:56:05,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:05,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:05,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681678565168, max timestamp=1681678565171Num uniques 859; 2023-04-16 20:56:05,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:05,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681678565168, max timestamp=1681678565171Num uniques 859; 2023-04-16 20:56:05,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:05,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681678565168, max timestamp=1681678565171Num uniques 859; 2023-04-16 20:56:05,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:05,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678625184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678625186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681678565182, max timestamp=1681678565189Num uniques 1068; 2023-04-16 20:56:05,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681678565182, max timestamp=1681678565189Num uniques 1068; 2023-04-16 20:56:05,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681678565182, max timestamp=1681678565189Num uniques 1068; 2023-04-16 20:56:05,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-16 20:56:05,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-16 20:56:05,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-16 20:56:05,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678625232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678625233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678625234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678625311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678625311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678625337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678625337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678625337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678625514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678625514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,524 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/664a1727d73440079db194b9c0ae50a4 2023-04-16 20:56:05,535 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/917d9c60cf664c7cb835db770b54b46d 2023-04-16 20:56:05,539 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/eb8013ea1e5a40bd8b04158ddd59af2e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/eb8013ea1e5a40bd8b04158ddd59af2e 2023-04-16 20:56:05,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678625539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678625541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678625541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,543 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/eb8013ea1e5a40bd8b04158ddd59af2e, entries=150, sequenceid=391, filesize=25.4 K 2023-04-16 20:56:05,544 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/664a1727d73440079db194b9c0ae50a4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/664a1727d73440079db194b9c0ae50a4 2023-04-16 20:56:05,547 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/664a1727d73440079db194b9c0ae50a4, entries=150, sequenceid=391, filesize=11.9 K 2023-04-16 20:56:05,547 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/917d9c60cf664c7cb835db770b54b46d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/917d9c60cf664c7cb835db770b54b46d 2023-04-16 20:56:05,550 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/917d9c60cf664c7cb835db770b54b46d, entries=150, sequenceid=391, filesize=11.9 K 2023-04-16 20:56:05,550 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=135.62 KB/138872 for 9853af3382d774c4b02dae6a5c604d40 in 466ms, sequenceid=391, compaction requested=true 2023-04-16 20:56:05,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:05,551 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:05,551 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:05,551 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:05,551 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,551 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:05,551 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:05,552 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,552 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:05,553 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,553 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:05,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:05,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@378b05aa[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:05,554 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:05,554 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,555 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,555 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:05,555 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:05,555 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:05,556 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,556 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:05,558 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:05,561 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:05,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:05,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:05,561 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:05,561 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:05,563 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,563 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:05,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,563 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:05,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:05,563 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:05,565 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:05,568 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:05,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:05,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:05,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,568 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:05,569 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:05,569 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:05,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:05,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:05,669 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:05,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:05,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:05,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:05,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:05,672 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:05,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:05,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:05,672 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:05,673 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,673 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,674 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:05,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:05,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,675 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,675 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:05,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:05,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:05,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:05,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:05,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:05,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:05,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:05,676 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:05,676 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:05,676 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:05,676 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=135.62 KB heapSize=382.70 KB 2023-04-16 20:56:05,676 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:05,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:05,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:05,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:05,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:05,677 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:05,692 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304169279198d680746c09d5d422865ebafe6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169279198d680746c09d5d422865ebafe6 2023-04-16 20:56:05,699 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=444, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/3fd29fca8db04faab62650a2f2269d05 2023-04-16 20:56:05,726 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/a8d0213f7bf44fafab184524477842d6 2023-04-16 20:56:05,748 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=41.69 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/e896d5cfbd4045bcb4eb91b96e62506a 2023-04-16 20:56:05,752 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/3fd29fca8db04faab62650a2f2269d05 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3fd29fca8db04faab62650a2f2269d05 2023-04-16 20:56:05,756 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3fd29fca8db04faab62650a2f2269d05, entries=150, sequenceid=444, filesize=25.4 K 2023-04-16 20:56:05,756 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/a8d0213f7bf44fafab184524477842d6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a8d0213f7bf44fafab184524477842d6 2023-04-16 20:56:05,760 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a8d0213f7bf44fafab184524477842d6, entries=150, sequenceid=444, filesize=11.9 K 2023-04-16 20:56:05,761 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/e896d5cfbd4045bcb4eb91b96e62506a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e896d5cfbd4045bcb4eb91b96e62506a 2023-04-16 20:56:05,765 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e896d5cfbd4045bcb4eb91b96e62506a, entries=150, sequenceid=444, filesize=11.9 K 2023-04-16 20:56:05,766 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~135.62 KB/138872, heapSize ~382.66 KB/391840, currentSize=0 B/0 for 9853af3382d774c4b02dae6a5c604d40 in 90ms, sequenceid=444, compaction requested=true 2023-04-16 20:56:05,766 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:05,766 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:05,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:05,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:05,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:05,766 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:05,769 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:05,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,769 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:05,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:05,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a2bc191[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:05,771 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:05,771 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,772 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,772 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:05,772 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:05,772 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:05,772 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,773 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:05,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:05,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:05,774 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:05,774 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:05,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:05,777 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:05,777 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:05,777 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:05,777 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:05,777 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:05,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:05,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,778 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:05,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:05,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:05,779 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,780 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:05,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,783 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:05,784 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:05,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:05,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:05,784 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:05,784 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:05,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:05,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:05,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,784 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:05,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:05,844 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:56:05,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:05,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:05,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:05,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:05,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:05,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:05,865 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041612545c76b9fa4835ba1cd68d11748738 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041612545c76b9fa4835ba1cd68d11748738 2023-04-16 20:56:05,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678565859, max timestamp=1681678565865Num uniques 859; 2023-04-16 20:56:05,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:05,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678565859, max timestamp=1681678565865Num uniques 859; 2023-04-16 20:56:05,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:05,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678565859, max timestamp=1681678565865Num uniques 859; 2023-04-16 20:56:05,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:05,884 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:05,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:05,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678625880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678625881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678625881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678625882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678625882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:05,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:05,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:05,888 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:05,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:05,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:05,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:05,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,888 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,889 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:05,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:05,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:05,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:05,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:05,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:05,891 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:05,891 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:05,891 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:05,891 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:05,891 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:05,892 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,892 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:05,892 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:05,893 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,893 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:05,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:05,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:05,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:05,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:05,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:05,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50dbf5c3[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:05,895 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:05,895 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,897 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:05,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:05,897 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:05,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,897 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:05,902 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,902 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:05,902 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:05,902 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:05,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:05,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:05,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:05,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:05,907 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:05,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:05,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:05,907 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:05,907 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:05,909 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,909 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:05,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,909 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,909 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:05,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,910 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:05,913 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,913 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:05,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:05,913 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:05,913 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:05,913 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:05,913 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:05,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:05,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:05,913 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:05,913 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:05,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:05,914 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,914 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,914 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:05,914 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:05,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:05,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678625986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678625986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678625986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678625986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:05,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:05,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678625987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,014 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:06,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:06,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:06,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:06,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:06,017 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:06,017 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,017 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:06,017 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:06,018 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,018 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,020 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,020 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,020 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,020 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:06,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:06,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:06,021 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:06,021 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,021 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:06,022 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,022 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:06,022 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:06,022 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:06,023 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:06,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26022d79[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:06,025 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:06,025 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,027 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:06,027 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:06,027 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:06,028 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,028 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:06,029 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:06,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:06,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:06,031 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:06,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:06,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:06,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:06,031 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:06,034 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,034 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,035 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,035 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,039 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,039 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,039 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,039 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,039 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,039 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,039 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:06,039 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:06,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:06,039 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,040 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,040 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:06,040 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:06,140 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:06,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:06,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:06,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:06,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:06,143 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:06,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:06,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:06,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,147 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,147 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,147 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,147 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:06,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:06,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:06,149 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:06,149 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,149 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:06,152 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,152 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:06,152 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:06,152 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:06,153 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,153 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,154 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,154 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:06,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cbca46[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:06,155 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:06,155 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,157 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,157 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:06,157 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:06,157 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:06,158 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,158 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:06,159 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:06,159 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,159 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,159 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:06,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:06,161 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:06,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:06,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:06,161 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:06,161 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:06,163 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,163 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,163 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,163 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,165 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,167 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:06,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:06,167 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,168 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:06,168 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:06,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:06,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678626189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678626189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678626190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678626190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678626191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,268 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:06,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:06,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:06,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:06,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:06,272 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:06,272 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,272 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:06,272 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:06,272 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,272 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,274 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,274 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,275 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:06,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:06,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:06,276 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:06,276 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,276 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:06,277 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,277 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:06,277 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:06,277 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:06,279 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,279 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,279 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,279 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:06,280 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=455, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/7eaecce4085e4fb9a0ca5e11e0e8401d 2023-04-16 20:56:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:06,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d68bda7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:06,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:06,282 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,283 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,283 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,283 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:06,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,283 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:06,283 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:06,283 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:06,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:06,285 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,286 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:06,286 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:06,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:06,289 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:06,289 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:06,289 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:06,289 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:06,289 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:06,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/05c4f82ef2ab41ee8fc7bbac88468295 2023-04-16 20:56:06,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,291 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,292 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,292 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,294 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,299 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:06,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:06,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,300 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:06,300 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:06,300 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:06,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,300 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,301 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/b6154ba414f9446d8adb39b9f38c6db6 2023-04-16 20:56:06,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/7eaecce4085e4fb9a0ca5e11e0e8401d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7eaecce4085e4fb9a0ca5e11e0e8401d 2023-04-16 20:56:06,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7eaecce4085e4fb9a0ca5e11e0e8401d, entries=200, sequenceid=455, filesize=32.3 K 2023-04-16 20:56:06,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/05c4f82ef2ab41ee8fc7bbac88468295 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/05c4f82ef2ab41ee8fc7bbac88468295 2023-04-16 20:56:06,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/05c4f82ef2ab41ee8fc7bbac88468295, entries=150, sequenceid=455, filesize=11.9 K 2023-04-16 20:56:06,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/b6154ba414f9446d8adb39b9f38c6db6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/b6154ba414f9446d8adb39b9f38c6db6 2023-04-16 20:56:06,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/b6154ba414f9446d8adb39b9f38c6db6, entries=150, sequenceid=455, filesize=11.9 K 2023-04-16 20:56:06,318 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 9853af3382d774c4b02dae6a5c604d40 in 474ms, sequenceid=455, compaction requested=true 2023-04-16 20:56:06,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:06,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:06,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:06,318 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:56:06,318 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 20:56:06,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:06,326 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:56:06,327 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:56:06,327 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,327 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/43ed3dc847ef440b8369dbc1aaa819fb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5bd8c2bb01c141548250b80f34ec9420, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/664a1727d73440079db194b9c0ae50a4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a8d0213f7bf44fafab184524477842d6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/05c4f82ef2ab41ee8fc7bbac88468295] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=60.2 K 2023-04-16 20:56:06,327 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 20:56:06,327 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:56:06,327 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,327 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2b92c60bd0494472b257320163258119, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/60537b2702924acbacccf9eafd6c5e63, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/eb8013ea1e5a40bd8b04158ddd59af2e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3fd29fca8db04faab62650a2f2269d05, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7eaecce4085e4fb9a0ca5e11e0e8401d] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=134.8 K 2023-04-16 20:56:06,327 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 43ed3dc847ef440b8369dbc1aaa819fb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681678563867 2023-04-16 20:56:06,328 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2b92c60bd0494472b257320163258119, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681678563867 2023-04-16 20:56:06,328 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5bd8c2bb01c141548250b80f34ec9420, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681678563917 2023-04-16 20:56:06,328 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 60537b2702924acbacccf9eafd6c5e63, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681678563917 2023-04-16 20:56:06,328 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 664a1727d73440079db194b9c0ae50a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681678564539 2023-04-16 20:56:06,330 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting eb8013ea1e5a40bd8b04158ddd59af2e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681678564539 2023-04-16 20:56:06,330 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting a8d0213f7bf44fafab184524477842d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681678565225 2023-04-16 20:56:06,330 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 3fd29fca8db04faab62650a2f2269d05, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681678565225 2023-04-16 20:56:06,330 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 05c4f82ef2ab41ee8fc7bbac88468295, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681678565839 2023-04-16 20:56:06,331 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7eaecce4085e4fb9a0ca5e11e0e8401d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681678565834 2023-04-16 20:56:06,341 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#625 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:06,346 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#624 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:06,359 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/2da2bd364f3a4ef391255f6df272e3b0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2da2bd364f3a4ef391255f6df272e3b0 2023-04-16 20:56:06,364 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into 2da2bd364f3a4ef391255f6df272e3b0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:06,364 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:06,365 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=11, startTime=1681678566318; duration=0sec 2023-04-16 20:56:06,365 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:06,365 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:56:06,366 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:56:06,366 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/C is initiating minor compaction (all files) 2023-04-16 20:56:06,367 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/C in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,367 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/51003de5977e4465ad1330b95d15070c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cf901dca34d8436590a5f6db99ed3302, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/206e14bf06a44445864681e3736b3070, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/917d9c60cf664c7cb835db770b54b46d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e896d5cfbd4045bcb4eb91b96e62506a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/b6154ba414f9446d8adb39b9f38c6db6] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=72.0 K 2023-04-16 20:56:06,367 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 51003de5977e4465ad1330b95d15070c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1681678563240 2023-04-16 20:56:06,367 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting cf901dca34d8436590a5f6db99ed3302, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681678563867 2023-04-16 20:56:06,368 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 206e14bf06a44445864681e3736b3070, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681678563917 2023-04-16 20:56:06,368 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 917d9c60cf664c7cb835db770b54b46d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681678564539 2023-04-16 20:56:06,368 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e896d5cfbd4045bcb4eb91b96e62506a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681678565225 2023-04-16 20:56:06,369 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting b6154ba414f9446d8adb39b9f38c6db6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681678565839 2023-04-16 20:56:06,377 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#C#compaction#626 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:06,384 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/8321063e190d4f40b96aed04d922ba92 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8321063e190d4f40b96aed04d922ba92 2023-04-16 20:56:06,388 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/C of 9853af3382d774c4b02dae6a5c604d40 into 8321063e190d4f40b96aed04d922ba92(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:06,388 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:06,388 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/C, priority=10, startTime=1681678566318; duration=0sec 2023-04-16 20:56:06,388 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:06,400 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:06,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:06,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:06,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:06,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:06,403 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:06,403 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,403 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:06,403 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:06,404 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,404 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,407 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,407 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,408 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:06,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:06,409 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:06,409 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,409 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:06,409 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:06,409 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:06,409 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,409 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:06,410 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-16 20:56:06,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:06,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:06,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:06,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:06,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:06,410 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:06,429 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304161a8767e04aba417283c94cadadf72a3f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161a8767e04aba417283c94cadadf72a3f 2023-04-16 20:56:06,434 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=492, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/191a56633f6b4b3d854728ba90062b90 2023-04-16 20:56:06,443 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/e6c62aeec51b4c29b207209c75936eb9 2023-04-16 20:56:06,453 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/2772978fc3ec444e9fe49c45a2a2ccde 2023-04-16 20:56:06,457 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/191a56633f6b4b3d854728ba90062b90 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/191a56633f6b4b3d854728ba90062b90 2023-04-16 20:56:06,461 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/191a56633f6b4b3d854728ba90062b90, entries=150, sequenceid=492, filesize=25.4 K 2023-04-16 20:56:06,461 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/e6c62aeec51b4c29b207209c75936eb9 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/e6c62aeec51b4c29b207209c75936eb9 2023-04-16 20:56:06,464 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/e6c62aeec51b4c29b207209c75936eb9, entries=150, sequenceid=492, filesize=11.9 K 2023-04-16 20:56:06,465 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/2772978fc3ec444e9fe49c45a2a2ccde as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2772978fc3ec444e9fe49c45a2a2ccde 2023-04-16 20:56:06,468 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2772978fc3ec444e9fe49c45a2a2ccde, entries=150, sequenceid=492, filesize=11.9 K 2023-04-16 20:56:06,469 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=0 B/0 for 9853af3382d774c4b02dae6a5c604d40 in 59ms, sequenceid=492, compaction requested=false 2023-04-16 20:56:06,469 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:06,469 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,469 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:06,469 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:06,469 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,469 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:06,469 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:06,471 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,471 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,471 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,472 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:06,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42122c24[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:06,473 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:06,473 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,475 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:06,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:06,475 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:06,475 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,475 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:06,477 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,477 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:06,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,478 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:06,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:06,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:06,480 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:06,480 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:06,480 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:06,480 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:06,480 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:06,482 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,482 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,482 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,483 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,484 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,486 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:06,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:06,486 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,487 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,487 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,487 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,487 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,487 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:06,487 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:06,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,487 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:06,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:06,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:56:06,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:06,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:06,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:06,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:06,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:06,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:06,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681678566532, max timestamp=1681678566541Num uniques 859; 2023-04-16 20:56:06,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678626540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681678566532, max timestamp=1681678566541Num uniques 859; 2023-04-16 20:56:06,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681678566532, max timestamp=1681678566541Num uniques 859; 2023-04-16 20:56:06,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 20:56:06,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 20:56:06,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 20:56:06,587 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:06,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:06,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:06,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:06,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:06,590 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:06,590 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,590 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:06,590 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:06,591 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,591 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,592 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,592 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,592 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,592 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:06,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:06,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:06,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:06,593 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:06,593 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,594 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:06,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678626590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678626590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678626591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678626596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,600 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:06,600 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,600 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:06,600 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:06,600 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:06,600 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,600 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:06,600 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:06,600 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,600 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:06,600 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:06,602 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,602 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,603 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,603 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:06,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75cca8fa[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:06,604 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:06,604 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,605 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,606 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:06,606 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:06,606 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:06,606 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,606 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:06,607 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,607 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:06,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,607 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,607 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:06,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:06,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:06,610 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:06,610 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:06,610 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:06,610 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:06,610 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:06,612 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,612 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,612 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,613 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,614 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:06,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:06,617 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,617 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:06,618 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:06,618 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:06,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678626644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678626697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678626697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678626697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678626700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,718 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:06,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:06,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:06,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:06,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:06,721 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:06,721 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,721 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:06,721 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:06,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,722 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,723 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,723 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,723 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:06,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:06,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:06,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:06,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:06,725 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:06,726 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,726 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:06,726 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:06,726 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:06,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:06,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:06,726 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:06,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,736 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:06,740 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:06,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62c2d5ff[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:06,743 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:06,743 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,745 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,745 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:06,745 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:06,745 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:06,745 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,745 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:06,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:06,747 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,747 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,748 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:06,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:06,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:06,751 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:06,751 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:06,751 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:06,751 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:06,751 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:06,754 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,754 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,754 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,755 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,759 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,764 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,765 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,765 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,765 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,765 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,765 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,765 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,765 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,765 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:06,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:06,765 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:06,765 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:06,772 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/7a9fb73a6ef8400eb91907db51f3feb7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7a9fb73a6ef8400eb91907db51f3feb7 2023-04-16 20:56:06,776 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into 7a9fb73a6ef8400eb91907db51f3feb7(size=26.3 K), total size for store is 51.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:06,776 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:06,776 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=11, startTime=1681678566318; duration=0sec 2023-04-16 20:56:06,776 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:06,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678626849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,865 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:06,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:06,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:06,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:06,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:06,869 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:06,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:06,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:06,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,869 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,871 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,871 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,872 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:06,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:06,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:06,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:06,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:06,873 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,873 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:06,873 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:06,873 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:06,873 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:06,875 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,875 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:06,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4fd985[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:06,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:06,876 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,877 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,877 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:06,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,878 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:06,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:06,879 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,879 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:06,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:06,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:06,881 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:06,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:06,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:06,881 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:06,881 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:06,888 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,888 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:06,889 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,888 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,890 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:06,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:06,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:06,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:06,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:06,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:06,892 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:06,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:06,893 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:06,892 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:06,893 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:06,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:06,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:06,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678626900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678626900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678626900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:06,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678626902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:06,935 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304167b0722c036714ac08b782162a393ec86 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167b0722c036714ac08b782162a393ec86 2023-04-16 20:56:06,993 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:06,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:06,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:06,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:06,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:06,998 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:06,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:06,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:06,998 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:06,999 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:06,999 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,000 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:07,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:07,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:07,000 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,001 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:07,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:07,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:07,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:07,002 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,002 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:07,002 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:07,002 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:07,004 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,004 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,004 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,005 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:07,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39efabe4[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:07,005 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:07,005 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,008 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:07,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,009 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:07,010 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,010 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,010 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:07,010 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:07,012 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:07,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:07,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:07,012 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:07,013 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:07,014 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,014 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,014 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:07,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:07,015 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,016 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:07,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:07,019 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:07,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:07,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:07,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:07,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,019 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:07,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,019 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:07,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:07,119 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:07,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:07,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:07,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:07,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:07,123 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:07,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:07,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:07,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:07,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,125 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:07,125 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:07,126 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,126 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:07,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:07,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:07,127 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,127 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:07,127 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:07,127 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:07,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:07,129 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:07,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:07,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f44f5b[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:07,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:07,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,132 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:07,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:07,132 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:07,132 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,132 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:07,133 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:07,133 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:07,133 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,133 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:07,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:07,136 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:07,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:07,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:07,136 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:07,136 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:07,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:07,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,138 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,140 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,141 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,142 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,142 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,142 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:07,142 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:07,142 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:07,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:07,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:07,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:07,142 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,142 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,142 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:07,142 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:07,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:07,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:07,142 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:07,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:07,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678627152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678627203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678627204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678627205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678627208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,242 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:07,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:07,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:07,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:07,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:07,246 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:07,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:07,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:07,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:07,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,246 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,250 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:07,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:07,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:07,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:07,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:07,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:07,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:07,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:07,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:07,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:07,252 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,252 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:07,252 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:07,253 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:07,253 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:07,254 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:07,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ded0ab7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:07,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:07,256 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,257 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,257 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:07,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,258 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:07,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:07,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:07,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:07,260 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,260 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:07,262 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:07,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:07,262 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:07,263 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:07,263 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:07,264 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,264 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,264 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:07,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,266 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,269 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:07,269 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:07,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:07,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:07,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:07,269 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:07,270 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:07,269 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,270 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,270 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,270 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,270 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,270 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:07,269 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:07,270 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:07,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:07,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:07,340 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=502, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/d9216693d2274e4abf2aaa2dc340e340 2023-04-16 20:56:07,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/2ef29875a14844ea8842dede2240c123 2023-04-16 20:56:07,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/c52e42c3057446dcb01d851e724d6fa6 2023-04-16 20:56:07,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/d9216693d2274e4abf2aaa2dc340e340 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d9216693d2274e4abf2aaa2dc340e340 2023-04-16 20:56:07,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d9216693d2274e4abf2aaa2dc340e340, entries=350, sequenceid=502, filesize=52.9 K 2023-04-16 20:56:07,370 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:07,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/2ef29875a14844ea8842dede2240c123 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2ef29875a14844ea8842dede2240c123 2023-04-16 20:56:07,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:07,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:07,373 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:07,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:07,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:07,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:07,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2ef29875a14844ea8842dede2240c123, entries=150, sequenceid=502, filesize=11.9 K 2023-04-16 20:56:07,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,374 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/c52e42c3057446dcb01d851e724d6fa6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c52e42c3057446dcb01d851e724d6fa6 2023-04-16 20:56:07,376 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:07,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:07,376 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,377 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,377 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:07,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:07,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:07,378 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,378 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:07,379 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:07,379 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,379 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:07,379 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:07,379 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,379 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:07,379 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:07,379 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c52e42c3057446dcb01d851e724d6fa6, entries=150, sequenceid=502, filesize=11.9 K 2023-04-16 20:56:07,380 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 9853af3382d774c4b02dae6a5c604d40 in 860ms, sequenceid=502, compaction requested=true 2023-04-16 20:56:07,380 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:07,380 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:07,380 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:07,380 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:07,380 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:56:07,380 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:56:07,381 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107226 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:56:07,381 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:56:07,381 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:56:07,381 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:56:07,381 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,381 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,381 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7a9fb73a6ef8400eb91907db51f3feb7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/191a56633f6b4b3d854728ba90062b90, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d9216693d2274e4abf2aaa2dc340e340] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=104.7 K 2023-04-16 20:56:07,381 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2da2bd364f3a4ef391255f6df272e3b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/e6c62aeec51b4c29b207209c75936eb9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2ef29875a14844ea8842dede2240c123] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=36.6 K 2023-04-16 20:56:07,381 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 7a9fb73a6ef8400eb91907db51f3feb7, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681678565839 2023-04-16 20:56:07,382 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 2da2bd364f3a4ef391255f6df272e3b0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681678565839 2023-04-16 20:56:07,382 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 191a56633f6b4b3d854728ba90062b90, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681678565878 2023-04-16 20:56:07,382 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,382 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,383 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting e6c62aeec51b4c29b207209c75936eb9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681678565878 2023-04-16 20:56:07,383 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d9216693d2274e4abf2aaa2dc340e340, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681678566509 2023-04-16 20:56:07,383 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,383 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,383 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 2ef29875a14844ea8842dede2240c123, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681678566510 2023-04-16 20:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@497a737e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:07,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:07,386 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,388 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:07,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:07,388 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:07,388 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,388 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:07,389 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,389 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,389 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:07,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:07,390 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#633 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:07,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:07,393 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:07,393 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:07,393 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:07,393 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:07,393 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:07,395 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:07,395 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,395 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:07,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,395 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:07,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:07,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:07,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:07,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,397 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:07,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:07,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,400 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:07,400 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:07,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:07,400 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:07,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:07,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:07,400 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:07,401 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:07,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:07,401 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#634 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:07,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:07,401 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:07,401 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:07,438 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/132d5ec25f5d4081bea78753e18ff2a8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/132d5ec25f5d4081bea78753e18ff2a8 2023-04-16 20:56:07,442 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into 132d5ec25f5d4081bea78753e18ff2a8(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:07,442 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:07,442 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=13, startTime=1681678567380; duration=0sec 2023-04-16 20:56:07,442 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:07,442 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:56:07,443 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:56:07,443 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/C is initiating minor compaction (all files) 2023-04-16 20:56:07,443 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/C in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,443 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8321063e190d4f40b96aed04d922ba92, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2772978fc3ec444e9fe49c45a2a2ccde, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c52e42c3057446dcb01d851e724d6fa6] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=36.5 K 2023-04-16 20:56:07,444 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8321063e190d4f40b96aed04d922ba92, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681678565839 2023-04-16 20:56:07,444 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 2772978fc3ec444e9fe49c45a2a2ccde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681678565878 2023-04-16 20:56:07,444 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting c52e42c3057446dcb01d851e724d6fa6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681678566510 2023-04-16 20:56:07,450 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#C#compaction#635 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:07,466 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/58ebc6aebdca46a48998eed253cca1f1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/58ebc6aebdca46a48998eed253cca1f1 2023-04-16 20:56:07,470 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/C of 9853af3382d774c4b02dae6a5c604d40 into 58ebc6aebdca46a48998eed253cca1f1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:07,471 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:07,471 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/C, priority=13, startTime=1681678567380; duration=0sec 2023-04-16 20:56:07,471 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:07,501 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:07,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:07,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:07,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:07,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:07,505 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:07,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:07,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:07,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:07,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,505 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,506 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:07,506 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:07,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:07,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,507 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:07,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:07,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:07,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:07,508 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:07,509 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:07,509 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-16 20:56:07,509 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:07,509 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:07,509 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:07,509 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:07,509 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:07,509 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:07,540 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416af7b542f67b44c3cb93cb91a2df4d4bb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416af7b542f67b44c3cb93cb91a2df4d4bb 2023-04-16 20:56:07,561 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=540, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/8313c971958a4a5e9e3d144ea6ea36eb 2023-04-16 20:56:07,586 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/3d25378aefb14d8fb8207839cb7cbb45 2023-04-16 20:56:07,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:07,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:07,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678627674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678627707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678627708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678627709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678627714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678627777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:07,810 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/ba99d2c7a5c749c1b844572a12bcf3ba as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ba99d2c7a5c749c1b844572a12bcf3ba 2023-04-16 20:56:07,813 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into ba99d2c7a5c749c1b844572a12bcf3ba(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:07,813 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:07,813 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=13, startTime=1681678567380; duration=0sec 2023-04-16 20:56:07,814 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:07,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:07,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678627980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,002 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/6cd98d43825947d9892998fa41198e34 2023-04-16 20:56:08,006 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/8313c971958a4a5e9e3d144ea6ea36eb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8313c971958a4a5e9e3d144ea6ea36eb 2023-04-16 20:56:08,009 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8313c971958a4a5e9e3d144ea6ea36eb, entries=150, sequenceid=540, filesize=25.4 K 2023-04-16 20:56:08,010 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/3d25378aefb14d8fb8207839cb7cbb45 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3d25378aefb14d8fb8207839cb7cbb45 2023-04-16 20:56:08,014 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3d25378aefb14d8fb8207839cb7cbb45, entries=150, sequenceid=540, filesize=11.9 K 2023-04-16 20:56:08,014 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/6cd98d43825947d9892998fa41198e34 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/6cd98d43825947d9892998fa41198e34 2023-04-16 20:56:08,018 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/6cd98d43825947d9892998fa41198e34, entries=150, sequenceid=540, filesize=11.9 K 2023-04-16 20:56:08,018 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 9853af3382d774c4b02dae6a5c604d40 in 509ms, sequenceid=540, compaction requested=false 2023-04-16 20:56:08,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:08,018 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:08,018 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:08,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:08,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:08,019 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:08,020 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d0e413e[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:08,022 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:08,022 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:08,024 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:08,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:08,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:08,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:08,031 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:08,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:08,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:08,031 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:08,031 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:08,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:08,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:08,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:08,033 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:08,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:08,033 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:08,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:08,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:08,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:08,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:08,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:08,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:08,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:08,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:08,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:08,037 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:08,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:08,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,038 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:08,038 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:08,038 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:08,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:08,038 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:08,138 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:08,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:08,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:08,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:08,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:08,141 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:08,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:08,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:08,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:08,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:08,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:08,143 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:08,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:08,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:08,143 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:08,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:08,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:08,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:08,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:08,145 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:08,145 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:08,146 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:56:08,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:08,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:08,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:08,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:08,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:08,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:08,173 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041625e102713ba64bfaa2d65830f32e80f8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041625e102713ba64bfaa2d65830f32e80f8 2023-04-16 20:56:08,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:08,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:08,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678568314, max timestamp=1681678568326Num uniques 859; 2023-04-16 20:56:08,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678568314, max timestamp=1681678568326Num uniques 859; 2023-04-16 20:56:08,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678568314, max timestamp=1681678568326Num uniques 859; 2023-04-16 20:56:08,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:08,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:08,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:08,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:08,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678628381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:08,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678628484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,586 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=551, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/d5c57bd97aa94d129a7ab85327649b57 2023-04-16 20:56:08,596 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/3fb94b5fe6ff47a1baa1afbb7fbd1e7e 2023-04-16 20:56:08,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:08,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678628686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:08,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678628712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:08,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678628713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:08,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678628717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:08,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678628723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:08,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:08,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678628989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,005 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/bffe3b5ff60b400b8f7cbc340a778596 2023-04-16 20:56:09,009 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/d5c57bd97aa94d129a7ab85327649b57 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d5c57bd97aa94d129a7ab85327649b57 2023-04-16 20:56:09,012 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d5c57bd97aa94d129a7ab85327649b57, entries=150, sequenceid=551, filesize=25.4 K 2023-04-16 20:56:09,013 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/3fb94b5fe6ff47a1baa1afbb7fbd1e7e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3fb94b5fe6ff47a1baa1afbb7fbd1e7e 2023-04-16 20:56:09,016 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3fb94b5fe6ff47a1baa1afbb7fbd1e7e, entries=150, sequenceid=551, filesize=11.9 K 2023-04-16 20:56:09,017 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/bffe3b5ff60b400b8f7cbc340a778596 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/bffe3b5ff60b400b8f7cbc340a778596 2023-04-16 20:56:09,020 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/bffe3b5ff60b400b8f7cbc340a778596, entries=150, sequenceid=551, filesize=11.9 K 2023-04-16 20:56:09,020 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9853af3382d774c4b02dae6a5c604d40 in 874ms, sequenceid=551, compaction requested=true 2023-04-16 20:56:09,020 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:09,020 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:09,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:09,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:09,020 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,021 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:09,021 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:09,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:09,022 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:09,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:09,023 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:09,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:09,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60b4953[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:09,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:09,024 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:09,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:09,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:09,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:09,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:09,028 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:09,028 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,028 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:09,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:09,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:09,029 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:09,029 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:09,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:09,032 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:09,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:09,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:09,032 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:09,032 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:09,034 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:09,034 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:09,034 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:09,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:09,034 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:09,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:09,035 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:09,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:09,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:09,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:09,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:09,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:09,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:09,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:09,038 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:09,038 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:09,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:09,038 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:09,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:09,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:09,038 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:09,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:09,138 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:09,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:09,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:09,141 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:09,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:09,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:09,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:09,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:09,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:09,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:09,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:09,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:09,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:09,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:09,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:09,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:09,146 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:09,146 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:09,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:09,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:09,146 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 20:56:09,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:09,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:09,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:09,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:09,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:09,146 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:09,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:09,494 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:09,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:09,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678629514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,561 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416659643ccd8da4fe3ad8e5061f12cdddd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416659643ccd8da4fe3ad8e5061f12cdddd 2023-04-16 20:56:09,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:09,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678629617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:09,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678629820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:09,965 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=588, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/e7b631ecdd9741fea8d3da9c6189b635 2023-04-16 20:56:09,975 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/f8d471d7c9794fb8994f23033fc60c2e 2023-04-16 20:56:10,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678630124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,384 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/600583335cfe4dba983a2b601e6654dc 2023-04-16 20:56:10,388 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/e7b631ecdd9741fea8d3da9c6189b635 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e7b631ecdd9741fea8d3da9c6189b635 2023-04-16 20:56:10,391 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e7b631ecdd9741fea8d3da9c6189b635, entries=150, sequenceid=588, filesize=25.4 K 2023-04-16 20:56:10,392 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/f8d471d7c9794fb8994f23033fc60c2e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f8d471d7c9794fb8994f23033fc60c2e 2023-04-16 20:56:10,396 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f8d471d7c9794fb8994f23033fc60c2e, entries=150, sequenceid=588, filesize=11.9 K 2023-04-16 20:56:10,397 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/600583335cfe4dba983a2b601e6654dc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/600583335cfe4dba983a2b601e6654dc 2023-04-16 20:56:10,400 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/600583335cfe4dba983a2b601e6654dc, entries=150, sequenceid=588, filesize=11.9 K 2023-04-16 20:56:10,400 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 9853af3382d774c4b02dae6a5c604d40 in 1254ms, sequenceid=588, compaction requested=true 2023-04-16 20:56:10,400 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:10,400 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:10,402 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:10,402 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:10,403 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:10,403 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:10,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:10,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:10,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33864397[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:10,404 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:10,405 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:10,406 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:10,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:10,406 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:10,406 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:10,406 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,406 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:10,406 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:10,407 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,407 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:10,407 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:10,408 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:10,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:10,410 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:10,410 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:10,411 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:10,411 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:10,411 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:10,418 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:10,418 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:10,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:10,418 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:10,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:10,419 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,420 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:10,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:10,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:10,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:10,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:10,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:10,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:10,423 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:10,423 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:10,423 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:10,422 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:10,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:10,422 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:10,423 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:10,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:10,523 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:10,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:10,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:10,527 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:10,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:10,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:10,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:10,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:10,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:10,528 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:10,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:10,529 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,529 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:10,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:10,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:10,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:10,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:10,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:10,531 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:10,531 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:10,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:10,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:10,532 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:10,532 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:10,532 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:10,532 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:56:10,532 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:10,532 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:10,532 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:10,532 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:10,532 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:10,532 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:10,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:10,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:10,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678570674, max timestamp=1681678570689Num uniques 859; 2023-04-16 20:56:10,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678570674, max timestamp=1681678570689Num uniques 859; 2023-04-16 20:56:10,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678570674, max timestamp=1681678570689Num uniques 859; 2023-04-16 20:56:10,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:10,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:10,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:10,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678630726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678630727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678630727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678630729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678630743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,748 DEBUG [Thread-4016] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:56:10,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678630830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678630831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678630831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:10,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678630833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:10,949 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304166207d93ff46d45a1b872b82bdc28b1f2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166207d93ff46d45a1b872b82bdc28b1f2 2023-04-16 20:56:10,962 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=598, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/4c4dcf72ebdb4612a327847419ae6f2b 2023-04-16 20:56:10,977 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/bb17dde083c64e3f95c216bbe4894d40 2023-04-16 20:56:10,986 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/7b6208ef6de0457d9020a09e7e41ecf1 2023-04-16 20:56:10,990 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/4c4dcf72ebdb4612a327847419ae6f2b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4c4dcf72ebdb4612a327847419ae6f2b 2023-04-16 20:56:10,994 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4c4dcf72ebdb4612a327847419ae6f2b, entries=100, sequenceid=598, filesize=18.6 K 2023-04-16 20:56:10,995 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/bb17dde083c64e3f95c216bbe4894d40 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb17dde083c64e3f95c216bbe4894d40 2023-04-16 20:56:10,998 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb17dde083c64e3f95c216bbe4894d40, entries=100, sequenceid=598, filesize=9.5 K 2023-04-16 20:56:10,999 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/7b6208ef6de0457d9020a09e7e41ecf1 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/7b6208ef6de0457d9020a09e7e41ecf1 2023-04-16 20:56:11,001 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/7b6208ef6de0457d9020a09e7e41ecf1, entries=100, sequenceid=598, filesize=9.5 K 2023-04-16 20:56:11,002 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9853af3382d774c4b02dae6a5c604d40 in 470ms, sequenceid=598, compaction requested=true 2023-04-16 20:56:11,002 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:11,002 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:11,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:11,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:11,002 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:11,004 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,004 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,004 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,004 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:11,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29fd271a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:11,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:11,006 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,009 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:11,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,010 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:11,011 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:11,013 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:11,014 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:11,014 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:11,014 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:11,014 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:11,015 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,015 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,015 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,016 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,017 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,019 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:11,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:11,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,019 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:11,019 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:11,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,019 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-16 20:56:11,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:11,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:11,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:11,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:11,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:11,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:11,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:11,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678631039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678631043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678631044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678631044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,119 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:11,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:11,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:11,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:11,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:11,122 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:11,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:11,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:11,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,123 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,125 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,125 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,125 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,126 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:11,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:11,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:11,126 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:11,126 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:11,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:11,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:11,127 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,127 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:11,127 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:11,127 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:11,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:11,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:11,127 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:11,128 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,128 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,129 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:11,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@560584aa[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:11,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:11,130 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,131 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,131 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:11,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:11,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:11,131 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:11,136 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:11,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,136 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:11,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:11,138 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:11,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:11,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:11,139 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:11,139 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:11,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,140 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,141 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,142 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,144 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:11,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,145 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,144 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:11,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:11,146 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:11,146 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:11,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678631145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678631148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678631148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678631149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,246 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:11,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:11,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:11,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:11,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:11,250 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:11,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:11,250 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:11,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,251 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,252 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,252 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:11,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:11,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:11,253 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:11,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:11,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:11,254 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:11,258 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,258 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:11,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a181ee8[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:11,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:11,259 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,260 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,261 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:11,261 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:11,261 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,261 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:11,261 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:11,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:11,262 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,262 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:11,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:11,265 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:11,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:11,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:11,265 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:11,265 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:11,266 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,266 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,266 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,267 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,268 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,271 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:11,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:11,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:11,271 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,272 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:11,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,272 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:11,272 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678631349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678631350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678631350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678631352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,372 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:11,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:11,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:11,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:11,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:11,375 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:11,376 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,376 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:11,376 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:11,376 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,376 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,378 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,378 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,378 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,378 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:11,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:11,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:11,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:11,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,390 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:11,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:11,392 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,392 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:11,392 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:11,392 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:11,392 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:11,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:11,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:11,393 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:11,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,395 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,395 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:11,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:11,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b837ec7[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:11,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:11,398 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,399 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,400 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:11,402 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,403 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:11,403 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:11,403 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:11,405 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,407 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:11,407 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,407 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:11,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:11,408 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:11,409 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:11,409 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:11,409 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:11,409 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:11,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,410 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,411 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,412 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,415 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:11,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:11,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,415 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,416 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:11,416 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:11,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:11,453 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416e28ad5a0f4eb4fb68754685687621c37 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e28ad5a0f4eb4fb68754685687621c37 2023-04-16 20:56:11,458 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=635, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/b35b8ef8de944c688e17c598905826fc 2023-04-16 20:56:11,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/0ad3031e9fc548a6af9e251e093edeb0 2023-04-16 20:56:11,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/05a4428badd84509a620e249702aaeee 2023-04-16 20:56:11,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/b35b8ef8de944c688e17c598905826fc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b35b8ef8de944c688e17c598905826fc 2023-04-16 20:56:11,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b35b8ef8de944c688e17c598905826fc, entries=200, sequenceid=635, filesize=32.3 K 2023-04-16 20:56:11,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/0ad3031e9fc548a6af9e251e093edeb0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0ad3031e9fc548a6af9e251e093edeb0 2023-04-16 20:56:11,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0ad3031e9fc548a6af9e251e093edeb0, entries=150, sequenceid=635, filesize=11.9 K 2023-04-16 20:56:11,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/05a4428badd84509a620e249702aaeee as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/05a4428badd84509a620e249702aaeee 2023-04-16 20:56:11,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/05a4428badd84509a620e249702aaeee, entries=150, sequenceid=635, filesize=11.9 K 2023-04-16 20:56:11,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 9853af3382d774c4b02dae6a5c604d40 in 459ms, sequenceid=635, compaction requested=true 2023-04-16 20:56:11,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:11,495 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:11,495 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:56:11,495 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:56:11,495 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:11,495 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:11,497 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 157290 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:56:11,497 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:56:11,497 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,497 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/132d5ec25f5d4081bea78753e18ff2a8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8313c971958a4a5e9e3d144ea6ea36eb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d5c57bd97aa94d129a7ab85327649b57, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e7b631ecdd9741fea8d3da9c6189b635, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4c4dcf72ebdb4612a327847419ae6f2b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b35b8ef8de944c688e17c598905826fc] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=153.6 K 2023-04-16 20:56:11,498 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 132d5ec25f5d4081bea78753e18ff2a8, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681678566510 2023-04-16 20:56:11,498 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71694 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:56:11,498 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:56:11,498 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,498 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ba99d2c7a5c749c1b844572a12bcf3ba, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3d25378aefb14d8fb8207839cb7cbb45, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3fb94b5fe6ff47a1baa1afbb7fbd1e7e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f8d471d7c9794fb8994f23033fc60c2e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb17dde083c64e3f95c216bbe4894d40, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0ad3031e9fc548a6af9e251e093edeb0] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=70.0 K 2023-04-16 20:56:11,498 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8313c971958a4a5e9e3d144ea6ea36eb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681678566554 2023-04-16 20:56:11,498 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ba99d2c7a5c749c1b844572a12bcf3ba, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681678566510 2023-04-16 20:56:11,499 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting d5c57bd97aa94d129a7ab85327649b57, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681678567668 2023-04-16 20:56:11,499 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3d25378aefb14d8fb8207839cb7cbb45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681678566554 2023-04-16 20:56:11,499 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting e7b631ecdd9741fea8d3da9c6189b635, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1681678568350 2023-04-16 20:56:11,499 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 3fb94b5fe6ff47a1baa1afbb7fbd1e7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681678567668 2023-04-16 20:56:11,500 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 4c4dcf72ebdb4612a327847419ae6f2b, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681678569504 2023-04-16 20:56:11,500 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting f8d471d7c9794fb8994f23033fc60c2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1681678568350 2023-04-16 20:56:11,500 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting b35b8ef8de944c688e17c598905826fc, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681678570725 2023-04-16 20:56:11,500 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting bb17dde083c64e3f95c216bbe4894d40, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681678569504 2023-04-16 20:56:11,501 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 0ad3031e9fc548a6af9e251e093edeb0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681678570725 2023-04-16 20:56:11,512 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#652 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:11,516 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:11,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:11,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:11,519 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#651 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:11,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:11,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:11,528 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:11,528 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,528 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:11,528 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:11,528 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,528 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,529 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,530 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:11,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:11,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:11,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:11,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:11,556 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:11,556 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,556 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:11,556 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:11,556 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,556 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:11,556 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:11,557 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-16 20:56:11,557 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:11,557 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:11,557 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:11,557 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:11,557 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:11,557 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:11,560 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/8fd66e14399e42ec95505e5596d8a7f6 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8fd66e14399e42ec95505e5596d8a7f6 2023-04-16 20:56:11,563 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into 8fd66e14399e42ec95505e5596d8a7f6(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:11,563 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:11,563 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=10, startTime=1681678571495; duration=0sec 2023-04-16 20:56:11,563 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:11,563 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 20:56:11,565 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71626 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 20:56:11,566 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/C is initiating minor compaction (all files) 2023-04-16 20:56:11,566 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/C in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,566 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/58ebc6aebdca46a48998eed253cca1f1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/6cd98d43825947d9892998fa41198e34, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/bffe3b5ff60b400b8f7cbc340a778596, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/600583335cfe4dba983a2b601e6654dc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/7b6208ef6de0457d9020a09e7e41ecf1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/05a4428badd84509a620e249702aaeee] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=69.9 K 2023-04-16 20:56:11,566 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 58ebc6aebdca46a48998eed253cca1f1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681678566510 2023-04-16 20:56:11,566 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 6cd98d43825947d9892998fa41198e34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681678566554 2023-04-16 20:56:11,567 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting bffe3b5ff60b400b8f7cbc340a778596, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681678567668 2023-04-16 20:56:11,567 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 600583335cfe4dba983a2b601e6654dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1681678568350 2023-04-16 20:56:11,568 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 7b6208ef6de0457d9020a09e7e41ecf1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1681678569504 2023-04-16 20:56:11,568 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 05a4428badd84509a620e249702aaeee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681678570725 2023-04-16 20:56:11,582 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#C#compaction#654 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:11,587 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/ed368f9261634c4dbd1ab1c48a9390a8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ed368f9261634c4dbd1ab1c48a9390a8 2023-04-16 20:56:11,591 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into ed368f9261634c4dbd1ab1c48a9390a8(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:11,591 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:11,591 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=10, startTime=1681678571495; duration=0sec 2023-04-16 20:56:11,591 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:11,597 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416e725a84772ba4e28bc01a660d222ecbe to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e725a84772ba4e28bc01a660d222ecbe 2023-04-16 20:56:11,609 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=644, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/63b602174d5048c9a5109fe68b02017a 2023-04-16 20:56:11,630 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/a32216deba7246ceb62a9fddbb9bbf55 2023-04-16 20:56:11,641 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/69392604f5dd4710b0ea8a9bbc4294b8 2023-04-16 20:56:11,645 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/63b602174d5048c9a5109fe68b02017a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/63b602174d5048c9a5109fe68b02017a 2023-04-16 20:56:11,648 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/63b602174d5048c9a5109fe68b02017a, entries=150, sequenceid=644, filesize=25.4 K 2023-04-16 20:56:11,649 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/a32216deba7246ceb62a9fddbb9bbf55 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a32216deba7246ceb62a9fddbb9bbf55 2023-04-16 20:56:11,652 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a32216deba7246ceb62a9fddbb9bbf55, entries=150, sequenceid=644, filesize=11.9 K 2023-04-16 20:56:11,652 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/69392604f5dd4710b0ea8a9bbc4294b8 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/69392604f5dd4710b0ea8a9bbc4294b8 2023-04-16 20:56:11,655 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/69392604f5dd4710b0ea8a9bbc4294b8, entries=150, sequenceid=644, filesize=11.9 K 2023-04-16 20:56:11,656 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 9853af3382d774c4b02dae6a5c604d40 in 99ms, sequenceid=644, compaction requested=false 2023-04-16 20:56:11,656 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:11,656 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:11,660 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:11,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,667 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,667 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b064d5a[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:11,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:11,669 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,670 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,670 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:11,670 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:11,670 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:11,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,670 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:11,672 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:11,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:11,674 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:11,674 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:11,675 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:11,675 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:11,675 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:11,676 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,676 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,676 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,677 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,678 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,683 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:11,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:11,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,683 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,684 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,684 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:11,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:11,684 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:11,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:11,703 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 20:56:11,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:11,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:11,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:11,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:11,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:11,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:11,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678631725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678631725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678571724, max timestamp=1681678571727Num uniques 859; 2023-04-16 20:56:11,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:11,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678571724, max timestamp=1681678571727Num uniques 859; 2023-04-16 20:56:11,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:11,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678571724, max timestamp=1681678571727Num uniques 859; 2023-04-16 20:56:11,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:11,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678631776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678631779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,784 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:11,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:11,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:11,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:11,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:11,787 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:11,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:11,787 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:11,788 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,788 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,789 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,789 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,789 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,790 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:11,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:11,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:11,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:11,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:11,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:11,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:11,792 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,792 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:11,792 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:11,792 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:11,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:11,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:11,792 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:11,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,794 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,794 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:11,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@315cb044[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:11,796 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:11,796 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,797 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:11,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:11,797 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:11,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,797 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:11,799 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,799 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,799 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,799 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:11,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:11,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:11,802 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:11,802 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:11,802 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:11,802 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:11,802 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:11,803 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,803 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,803 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,804 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,805 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,807 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,807 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,808 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:11,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:11,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,808 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,808 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:11,808 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:11,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678631830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678631830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678631881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:11,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678631882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,908 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:11,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:11,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:11,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:11,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:11,911 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:11,911 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,911 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:11,911 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:11,912 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,912 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,913 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,913 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,913 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,913 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:11,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:11,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:11,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:11,915 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,915 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:11,915 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:11,915 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:11,917 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,917 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,918 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:11,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b7907d9[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:11,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:11,919 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,920 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:11,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:11,920 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:11,921 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,921 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:11,921 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:11,922 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,922 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:11,924 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:11,924 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:11,924 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:11,924 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:11,924 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:11,926 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,926 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,926 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:11,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:11,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,927 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:11,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,931 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:11,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:11,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:11,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:11,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:11,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:11,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:11,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:11,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:11,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:11,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:11,932 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:11,932 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:11,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:11,997 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/fbf25c4f9b974921adf51f46b76f47bc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/fbf25c4f9b974921adf51f46b76f47bc 2023-04-16 20:56:12,001 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/C of 9853af3382d774c4b02dae6a5c604d40 into fbf25c4f9b974921adf51f46b76f47bc(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:12,002 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:12,002 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/C, priority=10, startTime=1681678571495; duration=0sec 2023-04-16 20:56:12,002 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:12,032 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:12,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:12,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:12,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:12,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:12,035 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:12,035 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:12,035 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:12,035 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:12,036 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,036 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:12,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678632034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,037 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,037 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:12,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678632035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:12,038 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,038 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:12,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:12,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:12,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:12,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:12,039 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:12,039 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:12,039 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:12,039 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:12,041 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,041 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:12,041 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:12,041 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:12,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:12,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b1c2b37[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:12,043 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:12,043 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:12,044 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:12,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:12,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:12,044 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:12,044 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:12,044 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:12,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,044 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:12,046 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:12,046 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,047 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:12,047 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:12,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:12,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:12,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:12,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:12,049 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:12,049 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:12,049 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:12,049 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:12,049 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:12,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:12,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,051 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:12,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:12,051 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,053 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:12,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:12,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:12,058 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:12,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:12,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:12,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:12,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:12,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:12,058 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:12,058 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:12,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:12,058 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:12,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678632086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678632086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,116 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304166bdc5a78baa5460098a442a383785bd5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166bdc5a78baa5460098a442a383785bd5 2023-04-16 20:56:12,121 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=657, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/da982e863e5844da9b2d99c279f8c03c 2023-04-16 20:56:12,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/11a6b56d13054fa6b9ee5edce68c5abc 2023-04-16 20:56:12,142 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/ea797382212e465cbdc8cf298565e032 2023-04-16 20:56:12,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/da982e863e5844da9b2d99c279f8c03c as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/da982e863e5844da9b2d99c279f8c03c 2023-04-16 20:56:12,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/da982e863e5844da9b2d99c279f8c03c, entries=300, sequenceid=657, filesize=46.1 K 2023-04-16 20:56:12,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/11a6b56d13054fa6b9ee5edce68c5abc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/11a6b56d13054fa6b9ee5edce68c5abc 2023-04-16 20:56:12,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/11a6b56d13054fa6b9ee5edce68c5abc, entries=150, sequenceid=657, filesize=11.9 K 2023-04-16 20:56:12,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/ea797382212e465cbdc8cf298565e032 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ea797382212e465cbdc8cf298565e032 2023-04-16 20:56:12,158 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:12,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:12,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:12,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ea797382212e465cbdc8cf298565e032, entries=150, sequenceid=657, filesize=11.9 K 2023-04-16 20:56:12,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:12,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:12,162 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:12,162 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:12,162 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9853af3382d774c4b02dae6a5c604d40 in 459ms, sequenceid=657, compaction requested=true 2023-04-16 20:56:12,162 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:12,162 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:12,162 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:12,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:12,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:12,162 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:56:12,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 20:56:12,162 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,162 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:12,163 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:56:12,165 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 100488 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:56:12,165 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:56:12,165 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:56:12,165 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:56:12,165 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:12,165 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:12,165 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:12,165 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8fd66e14399e42ec95505e5596d8a7f6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a32216deba7246ceb62a9fddbb9bbf55, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/11a6b56d13054fa6b9ee5edce68c5abc] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=36.9 K 2023-04-16 20:56:12,165 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ed368f9261634c4dbd1ab1c48a9390a8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/63b602174d5048c9a5109fe68b02017a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/da982e863e5844da9b2d99c279f8c03c] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=98.1 K 2023-04-16 20:56:12,165 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:12,165 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8fd66e14399e42ec95505e5596d8a7f6, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681678570725 2023-04-16 20:56:12,165 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:12,165 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:12,166 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ed368f9261634c4dbd1ab1c48a9390a8, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681678570725 2023-04-16 20:56:12,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:12,166 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting a32216deba7246ceb62a9fddbb9bbf55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1681678571042 2023-04-16 20:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:12,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:12,167 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 63b602174d5048c9a5109fe68b02017a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1681678571042 2023-04-16 20:56:12,167 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 11a6b56d13054fa6b9ee5edce68c5abc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681678571700 2023-04-16 20:56:12,167 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting da982e863e5844da9b2d99c279f8c03c, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681678571685 2023-04-16 20:56:12,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:12,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:12,168 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:12,168 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:12,168 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:12,181 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:12,181 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:12,181 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:12,181 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:12,181 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 20:56:12,181 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:12,182 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:12,182 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:12,182 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:12,182 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:12,182 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:12,203 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#660 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:12,221 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#662 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:12,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:12,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:12,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678632352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678632354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678632390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678632390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678632455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678632458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,618 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/83fa794c6dcb4f27a4c8c05437a29fc7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/83fa794c6dcb4f27a4c8c05437a29fc7 2023-04-16 20:56:12,621 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into 83fa794c6dcb4f27a4c8c05437a29fc7(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:12,622 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:12,622 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=13, startTime=1681678572162; duration=0sec 2023-04-16 20:56:12,622 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:12,622 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 20:56:12,623 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 20:56:12,623 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/C is initiating minor compaction (all files) 2023-04-16 20:56:12,623 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/C in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:12,623 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/fbf25c4f9b974921adf51f46b76f47bc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/69392604f5dd4710b0ea8a9bbc4294b8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ea797382212e465cbdc8cf298565e032] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=36.8 K 2023-04-16 20:56:12,623 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting fbf25c4f9b974921adf51f46b76f47bc, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681678570725 2023-04-16 20:56:12,624 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 69392604f5dd4710b0ea8a9bbc4294b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1681678571042 2023-04-16 20:56:12,624 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting ea797382212e465cbdc8cf298565e032, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681678571700 2023-04-16 20:56:12,626 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416264aec2b595348f6b539555a23684beb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416264aec2b595348f6b539555a23684beb 2023-04-16 20:56:12,629 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#C#compaction#663 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:12,636 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/15413fa2f2ff4e6aa155b1853705ea6a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/15413fa2f2ff4e6aa155b1853705ea6a 2023-04-16 20:56:12,639 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/C of 9853af3382d774c4b02dae6a5c604d40 into 15413fa2f2ff4e6aa155b1853705ea6a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:12,639 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:12,639 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/C, priority=13, startTime=1681678572162; duration=0sec 2023-04-16 20:56:12,639 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:12,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678632659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678632662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678632892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678632897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678632964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:12,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:12,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678632966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,032 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=694, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/4f3fa1d37de04c58a9b5ec5974484cdc 2023-04-16 20:56:13,045 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/8aa612e834f54dcea0429f3365cff1e4 2023-04-16 20:56:13,050 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/8b2f4e1e345a49da8fbc2cd4e0faefd4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b2f4e1e345a49da8fbc2cd4e0faefd4 2023-04-16 20:56:13,056 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into 8b2f4e1e345a49da8fbc2cd4e0faefd4(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:13,056 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:13,056 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=13, startTime=1681678572162; duration=0sec 2023-04-16 20:56:13,057 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:13,063 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/a90094462daa4b08836f5412b0eb074e 2023-04-16 20:56:13,067 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/4f3fa1d37de04c58a9b5ec5974484cdc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4f3fa1d37de04c58a9b5ec5974484cdc 2023-04-16 20:56:13,072 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4f3fa1d37de04c58a9b5ec5974484cdc, entries=150, sequenceid=694, filesize=25.4 K 2023-04-16 20:56:13,073 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/8aa612e834f54dcea0429f3365cff1e4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8aa612e834f54dcea0429f3365cff1e4 2023-04-16 20:56:13,076 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8aa612e834f54dcea0429f3365cff1e4, entries=150, sequenceid=694, filesize=11.9 K 2023-04-16 20:56:13,077 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/a90094462daa4b08836f5412b0eb074e as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/a90094462daa4b08836f5412b0eb074e 2023-04-16 20:56:13,081 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/a90094462daa4b08836f5412b0eb074e, entries=150, sequenceid=694, filesize=11.9 K 2023-04-16 20:56:13,082 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 9853af3382d774c4b02dae6a5c604d40 in 901ms, sequenceid=694, compaction requested=false 2023-04-16 20:56:13,082 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:13,082 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,084 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:13,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,085 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:13,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:13,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1209c793[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:13,088 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:13,088 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,090 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:13,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,090 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:13,092 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:13,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:13,096 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:13,096 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:13,096 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:13,096 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:13,096 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:13,102 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,102 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:13,102 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:13,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:13,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:13,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:13,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:13,104 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:13,106 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:13,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:13,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,109 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:13,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:13,110 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:13,110 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:13,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:13,110 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,110 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,110 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:13,110 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:13,211 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:13,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:13,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:13,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:13,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:13,214 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:13,214 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:13,214 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:13,214 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:13,214 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,214 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,216 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:13,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:13,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:13,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,216 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:13,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:13,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:13,218 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:13,218 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:13,218 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:13,218 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:13,218 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:13,218 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:13,218 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:13,218 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:56:13,219 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:13,219 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:13,219 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:13,219 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:13,219 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:13,219 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:13,268 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304164618b0dc99d04def9a3cf05b12aaaf69 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164618b0dc99d04def9a3cf05b12aaaf69 2023-04-16 20:56:13,275 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=708, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/8b7a5ae7b9cc48dca76bbc15cef6c0cf 2023-04-16 20:56:13,294 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/d23f01f5dfb04015a5ef730baed3dc61 2023-04-16 20:56:13,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:13,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:13,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678573491, max timestamp=1681678573499Num uniques 859; 2023-04-16 20:56:13,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:13,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678573491, max timestamp=1681678573499Num uniques 859; 2023-04-16 20:56:13,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:13,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681678573491, max timestamp=1681678573499Num uniques 859; 2023-04-16 20:56:13,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:13,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678633536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678633537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678633642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678633642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,706 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/de8af74a56ab45b09b21df43690bc0b0 2023-04-16 20:56:13,710 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/8b7a5ae7b9cc48dca76bbc15cef6c0cf as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b7a5ae7b9cc48dca76bbc15cef6c0cf 2023-04-16 20:56:13,713 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b7a5ae7b9cc48dca76bbc15cef6c0cf, entries=150, sequenceid=708, filesize=25.4 K 2023-04-16 20:56:13,714 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/d23f01f5dfb04015a5ef730baed3dc61 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d23f01f5dfb04015a5ef730baed3dc61 2023-04-16 20:56:13,716 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d23f01f5dfb04015a5ef730baed3dc61, entries=150, sequenceid=708, filesize=11.9 K 2023-04-16 20:56:13,717 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/de8af74a56ab45b09b21df43690bc0b0 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/de8af74a56ab45b09b21df43690bc0b0 2023-04-16 20:56:13,720 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/de8af74a56ab45b09b21df43690bc0b0, entries=150, sequenceid=708, filesize=11.9 K 2023-04-16 20:56:13,720 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9853af3382d774c4b02dae6a5c604d40 in 502ms, sequenceid=708, compaction requested=true 2023-04-16 20:56:13,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:13,721 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:13,721 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:13,721 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:13,721 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,721 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:13,721 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:13,723 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:13,723 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:13,724 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,724 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46039b06[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:13,725 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:13,725 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,727 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:13,727 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:13,728 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,728 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:13,729 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,729 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:13,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:13,731 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:13,731 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:13,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:13,732 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:13,732 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:13,733 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:13,733 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:13,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,733 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:13,734 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:13,735 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:13,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:13,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:13,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:13,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,738 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,739 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:13,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:13,739 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:13,739 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:13,739 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,739 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:13,739 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:13,839 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:13,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:13,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:13,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:13,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:13,842 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:13,842 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:13,842 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:13,843 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:13,843 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,843 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,845 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:13,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:13,845 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,846 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:13,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:13,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:13,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:13,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:13,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:13,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:13,847 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:13,847 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:13,847 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:13,847 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:13,847 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:13,847 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:13,847 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:13,847 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-16 20:56:13,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:13,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:13,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:13,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:13,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:13,848 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:13,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:13,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:13,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678633863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678633866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,889 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416bd8cdf8f90e143afad2799ab7435318f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bd8cdf8f90e143afad2799ab7435318f 2023-04-16 20:56:13,898 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=743, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/f32df6ef3faf43028586219ffba20800 2023-04-16 20:56:13,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678633904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,908 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/917b4b8e33154e75a36268c3c3240c28 2023-04-16 20:56:13,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678633908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678633967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:13,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:13,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678633970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:14,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678634169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:14,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678634174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,317 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/cb98152b721244e08200dac603abd321 2023-04-16 20:56:14,321 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/f32df6ef3faf43028586219ffba20800 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/f32df6ef3faf43028586219ffba20800 2023-04-16 20:56:14,325 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/f32df6ef3faf43028586219ffba20800, entries=150, sequenceid=743, filesize=25.4 K 2023-04-16 20:56:14,327 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/917b4b8e33154e75a36268c3c3240c28 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/917b4b8e33154e75a36268c3c3240c28 2023-04-16 20:56:14,332 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/917b4b8e33154e75a36268c3c3240c28, entries=150, sequenceid=743, filesize=11.9 K 2023-04-16 20:56:14,333 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/cb98152b721244e08200dac603abd321 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cb98152b721244e08200dac603abd321 2023-04-16 20:56:14,337 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cb98152b721244e08200dac603abd321, entries=150, sequenceid=743, filesize=11.9 K 2023-04-16 20:56:14,338 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 9853af3382d774c4b02dae6a5c604d40 in 491ms, sequenceid=743, compaction requested=true 2023-04-16 20:56:14,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:14,338 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:14,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:14,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:14,338 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,340 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:14,340 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:14,341 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:14,341 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75771bd0[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:14,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:14,343 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,344 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,344 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:14,344 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:14,345 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:14,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,345 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:14,346 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:14,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:14,346 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:14,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:14,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:14,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:14,349 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:14,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:14,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:14,349 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:14,349 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:14,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:14,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,352 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:14,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,353 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,354 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,357 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:14,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:14,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:14,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:14,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:14,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:14,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:14,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,357 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,357 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:14,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:14,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:14,357 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:14,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:14,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:14,458 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:14,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:14,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:14,461 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:14,461 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:14,461 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:14,461 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:14,462 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,462 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:14,465 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,465 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:14,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:14,466 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,466 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:14,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:14,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:14,467 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:14,467 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:14,468 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:56:14,468 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:14,468 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:14,468 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:14,468 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:14,469 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:14,469 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:14,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:14,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:14,488 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041602157a54b51b4b718c6bc5f0b664736a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041602157a54b51b4b718c6bc5f0b664736a 2023-04-16 20:56:14,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678574497, max timestamp=1681678574510Num uniques 859; 2023-04-16 20:56:14,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678574497, max timestamp=1681678574510Num uniques 859; 2023-04-16 20:56:14,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681678574497, max timestamp=1681678574510Num uniques 859; 2023-04-16 20:56:14,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:14,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:14,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 20:56:14,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:14,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678634534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:14,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678634536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:14,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678634638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:14,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678634640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:14,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40754 deadline: 1681678634756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,760 DEBUG [Thread-4016] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:56:14,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:14,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678634842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:14,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678634843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,893 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=754, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/74b8eb2c28224032802c2d6a5eb87f66 2023-04-16 20:56:14,901 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/948e95a0873e4f0fbcef134dd7c4b66b 2023-04-16 20:56:14,909 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/db783452d5b94df28429c4cdf2c74a21 2023-04-16 20:56:14,912 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/74b8eb2c28224032802c2d6a5eb87f66 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/74b8eb2c28224032802c2d6a5eb87f66 2023-04-16 20:56:14,915 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/74b8eb2c28224032802c2d6a5eb87f66, entries=150, sequenceid=754, filesize=25.4 K 2023-04-16 20:56:14,916 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/948e95a0873e4f0fbcef134dd7c4b66b as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/948e95a0873e4f0fbcef134dd7c4b66b 2023-04-16 20:56:14,919 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/948e95a0873e4f0fbcef134dd7c4b66b, entries=150, sequenceid=754, filesize=11.9 K 2023-04-16 20:56:14,919 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/db783452d5b94df28429c4cdf2c74a21 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/db783452d5b94df28429c4cdf2c74a21 2023-04-16 20:56:14,922 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/db783452d5b94df28429c4cdf2c74a21, entries=150, sequenceid=754, filesize=11.9 K 2023-04-16 20:56:14,923 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9853af3382d774c4b02dae6a5c604d40 in 455ms, sequenceid=754, compaction requested=true 2023-04-16 20:56:14,923 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:14,923 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:14,923 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:14,926 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,926 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:14,927 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,927 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:14,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:14,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:14,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:14,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f38c3da[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:14,928 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:14,928 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,929 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,930 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:14,930 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:14,930 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:14,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,930 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:14,931 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:14,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:14,931 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:14,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:14,934 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:14,934 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:14,934 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:14,934 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:14,934 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:14,935 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:14,935 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,936 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:14,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:14,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:14,936 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,937 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:14,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:14,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:14,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:14,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:14,941 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:14,940 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:14,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:14,941 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:14,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:14,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:14,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:14,941 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:14,941 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:15,041 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:15,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:15,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:15,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:15,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:15,045 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:15,045 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:15,045 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:15,045 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:15,045 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,045 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:15,046 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,046 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:15,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:15,047 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,047 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:15,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:15,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:15,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:15,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:15,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:15,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:15,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:15,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:15,048 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:15,048 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:15,048 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:15,048 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-16 20:56:15,048 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:15,049 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:15,049 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:15,049 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:15,049 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:15,049 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 20:56:15,069 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416872cd7a2fd114be68bd877c428f0bc59 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416872cd7a2fd114be68bd877c428f0bc59 2023-04-16 20:56:15,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:15,147 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:15,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678635162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678635165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678635266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678635268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678635470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678635471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,488 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=789, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/0dcceecf38694396b32fb5192370f8ba 2023-04-16 20:56:15,498 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/d1ba427eee644a319fb787b44504c9ca 2023-04-16 20:56:15,508 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/5196bff1881441edbd7d03338ff7aa92 2023-04-16 20:56:15,512 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/0dcceecf38694396b32fb5192370f8ba as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/0dcceecf38694396b32fb5192370f8ba 2023-04-16 20:56:15,515 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/0dcceecf38694396b32fb5192370f8ba, entries=150, sequenceid=789, filesize=25.4 K 2023-04-16 20:56:15,516 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/d1ba427eee644a319fb787b44504c9ca as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1ba427eee644a319fb787b44504c9ca 2023-04-16 20:56:15,519 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1ba427eee644a319fb787b44504c9ca, entries=150, sequenceid=789, filesize=11.9 K 2023-04-16 20:56:15,520 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/5196bff1881441edbd7d03338ff7aa92 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/5196bff1881441edbd7d03338ff7aa92 2023-04-16 20:56:15,523 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/5196bff1881441edbd7d03338ff7aa92, entries=150, sequenceid=789, filesize=11.9 K 2023-04-16 20:56:15,523 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 9853af3382d774c4b02dae6a5c604d40 in 475ms, sequenceid=789, compaction requested=true 2023-04-16 20:56:15,523 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:15,523 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:15,524 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:15,525 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,525 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:15,525 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:15,525 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d1a6f1[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:15,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:15,527 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:15,528 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:15,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:15,528 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:15,528 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:15,528 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:15,529 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,529 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:15,530 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:15,530 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:15,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:15,533 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:15,533 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:15,533 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:15,533 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:15,533 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:15,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:15,535 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,535 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:15,536 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:15,538 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:15,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:15,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:15,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:15,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:15,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:15,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:15,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:15,539 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,539 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:15,539 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:15,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:15,539 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:15,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:15,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:15,639 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:15,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:15,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 20:56:15,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 20:56:15,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 20:56:15,643 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 20:56:15,643 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:15,643 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 20:56:15,643 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 20:56:15,643 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,643 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:15,645 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:15,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:15,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:15,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:15,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,645 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 20:56:15,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:15,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 20:56:15,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:15,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 20:56:15,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 20:56:15,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 20:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 20:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 20:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 20:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 20:56:15,646 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:15,646 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. started... 2023-04-16 20:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 20:56:15,647 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 20:56:15,647 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:15,647 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:15,647 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:15,647 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:15,647 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:15,647 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:15,665 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416639c54d8e045410a8db055a53039d14f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416639c54d8e045410a8db055a53039d14f 2023-04-16 20:56:15,670 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=800, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/ebad7e17c5534d29b5e5865f9076086a 2023-04-16 20:56:15,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:15,778 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. as already flushing 2023-04-16 20:56:15,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678575802, max timestamp=1681678575815Num uniques 859; 2023-04-16 20:56:15,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:15,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678575802, max timestamp=1681678575815Num uniques 859; 2023-04-16 20:56:15,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681678575802, max timestamp=1681678575815Num uniques 859; 2023-04-16 20:56:15,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:15,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 20:56:15,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678635863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41211] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678635864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40784 deadline: 1681678635920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,925 DEBUG [Thread-4018] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:56:15,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41211] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40748 deadline: 1681678635930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,934 DEBUG [Thread-4012] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., hostname=jenkins-hbase4.apache.org,41211,1681678416093, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 20:56:15,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678635968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:15,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:15,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678635968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,082 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/be8eb4dadb48441992556b5e8b0b06cb 2023-04-16 20:56:16,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:16,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678636172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:16,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678636172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,201 DEBUG [Thread-4023] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:54578 2023-04-16 20:56:16,201 DEBUG [Thread-4023] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:16,201 DEBUG [Thread-4025] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eedd898 to 127.0.0.1:54578 2023-04-16 20:56:16,201 DEBUG [Thread-4025] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:16,203 DEBUG [Thread-4027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x618c0022 to 127.0.0.1:54578 2023-04-16 20:56:16,204 DEBUG [Thread-4027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:16,204 DEBUG [Thread-4029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d33872d to 127.0.0.1:54578 2023-04-16 20:56:16,204 DEBUG [Thread-4029] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:16,206 DEBUG [Thread-4021] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:54578 2023-04-16 20:56:16,206 DEBUG [Thread-4021] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:16,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:16,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40750 deadline: 1681678636476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 20:56:16,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40762 deadline: 1681678636477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,491 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/c71d5f09bbe540daa925e3ebe5da4c46 2023-04-16 20:56:16,494 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/ebad7e17c5534d29b5e5865f9076086a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ebad7e17c5534d29b5e5865f9076086a 2023-04-16 20:56:16,497 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ebad7e17c5534d29b5e5865f9076086a, entries=100, sequenceid=800, filesize=18.6 K 2023-04-16 20:56:16,497 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/be8eb4dadb48441992556b5e8b0b06cb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/be8eb4dadb48441992556b5e8b0b06cb 2023-04-16 20:56:16,501 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/be8eb4dadb48441992556b5e8b0b06cb, entries=100, sequenceid=800, filesize=9.5 K 2023-04-16 20:56:16,501 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/c71d5f09bbe540daa925e3ebe5da4c46 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c71d5f09bbe540daa925e3ebe5da4c46 2023-04-16 20:56:16,503 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c71d5f09bbe540daa925e3ebe5da4c46, entries=100, sequenceid=800, filesize=9.5 K 2023-04-16 20:56:16,504 INFO [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9853af3382d774c4b02dae6a5c604d40 in 857ms, sequenceid=800, compaction requested=true 2023-04-16 20:56:16,504 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:16,504 DEBUG [rs(jenkins-hbase4.apache.org,41211,1681678416093)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 20:56:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 20:56:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 20:56:16,504 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 20:56:16,508 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:16,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:16,508 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 20:56:16,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:16,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,41211,1681678416093' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 20:56:16,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7950c226[Count = 0] remaining members to acquire global barrier 2023-04-16 20:56:16,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 20:56:16,509 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:16,510 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:16,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:16,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:16,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 20:56:16,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 20:56:16,510 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,41211,1681678416093' in zk 2023-04-16 20:56:16,510 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,510 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 20:56:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 20:56:16,511 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:16,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:16,511 DEBUG [member: 'jenkins-hbase4.apache.org,41211,1681678416093' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 20:56:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,41211,1681678416093': 2023-04-16 20:56:16,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,41211,1681678416093' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 20:56:16,513 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 20:56:16,513 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 20:56:16,513 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 20:56:16,513 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 20:56:16,513 INFO [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 20:56:16,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:16,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 20:56:16,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:16,515 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 20:56:16,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 20:56:16,515 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 20:56:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,516 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 20:56:16,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 20:56:16,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 20:56:16,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:16,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 20:56:16,519 DEBUG [(jenkins-hbase4.apache.org,46091,1681678414874)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 20:56:16,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 20:56:16,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 20:56:16,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 20:56:16,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:16,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:16,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 20:56:16,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 20:56:16,519 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 20:56:16,520 DEBUG [Thread-4020] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 20:56:16,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 20:56:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 20:56:16,520 DEBUG [Thread-4020] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 20:56:16,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 20:56:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 20:56:16,620 DEBUG [Thread-4020] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 20:56:16,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 20:56:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41211] regionserver.HRegion(9158): Flush requested on 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:16,981 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-16 20:56:16,981 DEBUG [Thread-4014] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:54578 2023-04-16 20:56:16,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:16,981 DEBUG [Thread-4014] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:16,981 DEBUG [Thread-4010] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:54578 2023-04-16 20:56:16,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:56:16,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:16,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:56:16,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:16,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 20:56:16,981 DEBUG [Thread-4010] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:17,394 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416199e3e79d1d541529fe12953ecbf5629 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416199e3e79d1d541529fe12953ecbf5629 2023-04-16 20:56:17,798 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=838, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/6c0dc19f0de247ebb0c2f2f75f695543 2023-04-16 20:56:17,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/55a072de59fd43f7ad5b1551a68eae89 2023-04-16 20:56:17,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/668982571b23452eaefe18bf714578ba 2023-04-16 20:56:17,815 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/6c0dc19f0de247ebb0c2f2f75f695543 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/6c0dc19f0de247ebb0c2f2f75f695543 2023-04-16 20:56:17,819 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/6c0dc19f0de247ebb0c2f2f75f695543, entries=150, sequenceid=838, filesize=25.4 K 2023-04-16 20:56:17,820 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/55a072de59fd43f7ad5b1551a68eae89 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/55a072de59fd43f7ad5b1551a68eae89 2023-04-16 20:56:17,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/55a072de59fd43f7ad5b1551a68eae89, entries=150, sequenceid=838, filesize=11.9 K 2023-04-16 20:56:17,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/668982571b23452eaefe18bf714578ba as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/668982571b23452eaefe18bf714578ba 2023-04-16 20:56:17,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/668982571b23452eaefe18bf714578ba, entries=150, sequenceid=838, filesize=11.9 K 2023-04-16 20:56:17,828 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=0 B/0 for 9853af3382d774c4b02dae6a5c604d40 in 847ms, sequenceid=838, compaction requested=true 2023-04-16 20:56:17,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:17,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:17,828 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-16 20:56:17,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:17,828 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-16 20:56:17,828 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:17,838 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 202648 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-16 20:56:17,839 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/A is initiating minor compaction (all files) 2023-04-16 20:56:17,839 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96372 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-16 20:56:17,839 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/A in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:17,839 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/B is initiating minor compaction (all files) 2023-04-16 20:56:17,839 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/B in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:17,839 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b2f4e1e345a49da8fbc2cd4e0faefd4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4f3fa1d37de04c58a9b5ec5974484cdc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b7a5ae7b9cc48dca76bbc15cef6c0cf, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/f32df6ef3faf43028586219ffba20800, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/74b8eb2c28224032802c2d6a5eb87f66, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/0dcceecf38694396b32fb5192370f8ba, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ebad7e17c5534d29b5e5865f9076086a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/6c0dc19f0de247ebb0c2f2f75f695543] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=197.9 K 2023-04-16 20:56:17,839 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/83fa794c6dcb4f27a4c8c05437a29fc7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8aa612e834f54dcea0429f3365cff1e4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d23f01f5dfb04015a5ef730baed3dc61, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/917b4b8e33154e75a36268c3c3240c28, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/948e95a0873e4f0fbcef134dd7c4b66b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1ba427eee644a319fb787b44504c9ca, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/be8eb4dadb48441992556b5e8b0b06cb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/55a072de59fd43f7ad5b1551a68eae89] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=94.1 K 2023-04-16 20:56:17,839 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8b2f4e1e345a49da8fbc2cd4e0faefd4, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681678571700 2023-04-16 20:56:17,839 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 83fa794c6dcb4f27a4c8c05437a29fc7, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681678571700 2023-04-16 20:56:17,840 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 4f3fa1d37de04c58a9b5ec5974484cdc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1681678571771 2023-04-16 20:56:17,840 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 8aa612e834f54dcea0429f3365cff1e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1681678571771 2023-04-16 20:56:17,840 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 8b7a5ae7b9cc48dca76bbc15cef6c0cf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1681678572349 2023-04-16 20:56:17,840 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d23f01f5dfb04015a5ef730baed3dc61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1681678572349 2023-04-16 20:56:17,840 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting f32df6ef3faf43028586219ffba20800, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1681678573531 2023-04-16 20:56:17,841 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 917b4b8e33154e75a36268c3c3240c28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1681678573531 2023-04-16 20:56:17,841 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 74b8eb2c28224032802c2d6a5eb87f66, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1681678573860 2023-04-16 20:56:17,841 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 948e95a0873e4f0fbcef134dd7c4b66b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1681678573860 2023-04-16 20:56:17,841 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 0dcceecf38694396b32fb5192370f8ba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1681678574532 2023-04-16 20:56:17,841 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting d1ba427eee644a319fb787b44504c9ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1681678574532 2023-04-16 20:56:17,841 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting ebad7e17c5534d29b5e5865f9076086a, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681678575161 2023-04-16 20:56:17,841 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting be8eb4dadb48441992556b5e8b0b06cb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681678575161 2023-04-16 20:56:17,842 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] compactions.Compactor(207): Compacting 6c0dc19f0de247ebb0c2f2f75f695543, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1681678575862 2023-04-16 20:56:17,842 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 55a072de59fd43f7ad5b1551a68eae89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1681678575862 2023-04-16 20:56:17,855 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#B#compaction#685 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:17,861 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#A#compaction#684 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:17,865 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/d269d20521b34f1289f00ea6bcbfe7a4 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d269d20521b34f1289f00ea6bcbfe7a4 2023-04-16 20:56:17,868 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/B of 9853af3382d774c4b02dae6a5c604d40 into d269d20521b34f1289f00ea6bcbfe7a4(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:17,868 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:17,869 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/B, priority=8, startTime=1681678577828; duration=0sec 2023-04-16 20:56:17,869 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 20:56:17,869 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-16 20:56:17,871 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96304 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-16 20:56:17,871 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1912): 9853af3382d774c4b02dae6a5c604d40/C is initiating minor compaction (all files) 2023-04-16 20:56:17,871 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9853af3382d774c4b02dae6a5c604d40/C in TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:17,871 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/15413fa2f2ff4e6aa155b1853705ea6a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/a90094462daa4b08836f5412b0eb074e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/de8af74a56ab45b09b21df43690bc0b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cb98152b721244e08200dac603abd321, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/db783452d5b94df28429c4cdf2c74a21, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/5196bff1881441edbd7d03338ff7aa92, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c71d5f09bbe540daa925e3ebe5da4c46, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/668982571b23452eaefe18bf714578ba] into tmpdir=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp, totalSize=94.0 K 2023-04-16 20:56:17,871 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 15413fa2f2ff4e6aa155b1853705ea6a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681678571700 2023-04-16 20:56:17,872 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting a90094462daa4b08836f5412b0eb074e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1681678571771 2023-04-16 20:56:17,872 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting de8af74a56ab45b09b21df43690bc0b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1681678572349 2023-04-16 20:56:17,872 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting cb98152b721244e08200dac603abd321, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1681678573531 2023-04-16 20:56:17,873 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting db783452d5b94df28429c4cdf2c74a21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1681678573860 2023-04-16 20:56:17,873 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 5196bff1881441edbd7d03338ff7aa92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1681678574532 2023-04-16 20:56:17,873 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting c71d5f09bbe540daa925e3ebe5da4c46, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681678575161 2023-04-16 20:56:17,873 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] compactions.Compactor(207): Compacting 668982571b23452eaefe18bf714578ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1681678575862 2023-04-16 20:56:17,882 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] throttle.PressureAwareThroughputController(145): 9853af3382d774c4b02dae6a5c604d40#C#compaction#686 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 20:56:18,289 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/fcf51076f6314d9c91aed93579243359 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/fcf51076f6314d9c91aed93579243359 2023-04-16 20:56:18,292 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/C of 9853af3382d774c4b02dae6a5c604d40 into fcf51076f6314d9c91aed93579243359(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:18,292 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:18,292 INFO [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/C, priority=8, startTime=1681678577828; duration=0sec 2023-04-16 20:56:18,292 DEBUG [RS:0;jenkins-hbase4:41211-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:18,680 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/4b45eefc0fd849bfaec0809b81785e1a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4b45eefc0fd849bfaec0809b81785e1a 2023-04-16 20:56:18,684 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9853af3382d774c4b02dae6a5c604d40/A of 9853af3382d774c4b02dae6a5c604d40 into 4b45eefc0fd849bfaec0809b81785e1a(size=27.0 K), total size for store is 27.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 20:56:18,684 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:18,684 INFO [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40., storeName=9853af3382d774c4b02dae6a5c604d40/A, priority=8, startTime=1681678577828; duration=0sec 2023-04-16 20:56:18,684 DEBUG [RS:0;jenkins-hbase4:41211-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 20:56:19,937 DEBUG [Thread-4018] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:54578 2023-04-16 20:56:19,937 DEBUG [Thread-4018] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:19,969 DEBUG [Thread-4012] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:54578 2023-04-16 20:56:19,969 DEBUG [Thread-4012] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:24,769 DEBUG [Thread-4016] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:54578 2023-04-16 20:56:24,769 DEBUG [Thread-4016] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:24,769 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 20:56:24,769 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-04-16 20:56:24,769 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 184 2023-04-16 20:56:24,769 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 165 2023-04-16 20:56:24,769 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(394): wrote 104 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 4404 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 4309 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 4283 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 4400 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(398): read 4291 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 20:56:24,770 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:54578 2023-04-16 20:56:24,770 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:24,770 INFO [Listener at localhost/45179] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 20:56:24,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 20:56:24,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 20:56:24,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 20:56:24,774 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678584773"}]},"ts":"1681678584773"} 2023-04-16 20:56:24,774 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 20:56:24,776 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 20:56:24,776 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, UNASSIGN}] 2023-04-16 20:56:24,777 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, UNASSIGN 2023-04-16 20:56:24,778 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=9853af3382d774c4b02dae6a5c604d40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:24,778 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678584778"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681678584778"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681678584778"}]},"ts":"1681678584778"} 2023-04-16 20:56:24,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093}] 2023-04-16 20:56:24,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 20:56:24,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:24,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9853af3382d774c4b02dae6a5c604d40, disabling compactions & flushes 2023-04-16 20:56:24,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:24,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:24,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. after waiting 0 ms 2023-04-16 20:56:24,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:24,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9853af3382d774c4b02dae6a5c604d40 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-16 20:56:24,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=A 2023-04-16 20:56:24,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:24,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=B 2023-04-16 20:56:24,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:24,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9853af3382d774c4b02dae6a5c604d40, store=C 2023-04-16 20:56:24,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 20:56:25,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 20:56:25,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041621c7473c035c4ad1ab177ac800b54e1b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041621c7473c035c4ad1ab177ac800b54e1b 2023-04-16 20:56:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 20:56:25,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=847, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/7f553d2e50d449bb9be97f75f9c04e1d 2023-04-16 20:56:25,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=847 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/d1cdacd3ecf2439f89eb3b5ee67eb79a 2023-04-16 20:56:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 20:56:26,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=847 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/d19522945082449eac4a165f988e23cc 2023-04-16 20:56:26,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/A/7f553d2e50d449bb9be97f75f9c04e1d as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7f553d2e50d449bb9be97f75f9c04e1d 2023-04-16 20:56:26,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7f553d2e50d449bb9be97f75f9c04e1d, entries=50, sequenceid=847, filesize=11.7 K 2023-04-16 20:56:26,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/B/d1cdacd3ecf2439f89eb3b5ee67eb79a as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1cdacd3ecf2439f89eb3b5ee67eb79a 2023-04-16 20:56:26,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1cdacd3ecf2439f89eb3b5ee67eb79a, entries=50, sequenceid=847, filesize=7.1 K 2023-04-16 20:56:26,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/.tmp/C/d19522945082449eac4a165f988e23cc as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d19522945082449eac4a165f988e23cc 2023-04-16 20:56:26,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d19522945082449eac4a165f988e23cc, entries=50, sequenceid=847, filesize=7.1 K 2023-04-16 20:56:26,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 9853af3382d774c4b02dae6a5c604d40 in 1240ms, sequenceid=847, compaction requested=false 2023-04-16 20:56:26,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2704a0642d9a46e9a8c213f04e042334, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e502e329789a42219dd2453516f8edf3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/32900ffa448042b096c346e1c8a87a79, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2675171c9ed7406e8fb4184926925ac2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3aa158af4c21488e9c6af541fa26f2b6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ec3bba95fab74452a9e1249d94deeff0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/9d9e036edd414faf93955a3bffb5d23a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e6abc28f59344518bf95b9ec6f27066b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8622eb9a357948d6ac0b3d14b7ee0bff, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/102a79efb4634694a7fe07d970d74df5, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c9a95d1ea7bc49f4a9e57d6666408314, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c2e53985332c4d2f90055e53ca30407f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ddb9ccf568564db2abbd4943da7c8324, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b630a135cfe44963b749c0ad452a2280, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/a65af645b2d54fcf83027ebfed929062, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/86ee031a103745c38fa6d7471a639bd9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c96838fe89eb40d98abdf27cf3aa4193, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3f511669c2af4ea0b3f5c03dcfb7640b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c21b67d588d0416cb8e422fe13d8889e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/95da552ee04e415b9604801ca84d9ed6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7ca748abe6064bf7af8f48a688151b71, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/5b6a5a3dcda547d6b56feee69072111e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2b92c60bd0494472b257320163258119, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/60537b2702924acbacccf9eafd6c5e63, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/eb8013ea1e5a40bd8b04158ddd59af2e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3fd29fca8db04faab62650a2f2269d05, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7eaecce4085e4fb9a0ca5e11e0e8401d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7a9fb73a6ef8400eb91907db51f3feb7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/191a56633f6b4b3d854728ba90062b90, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d9216693d2274e4abf2aaa2dc340e340, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/132d5ec25f5d4081bea78753e18ff2a8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8313c971958a4a5e9e3d144ea6ea36eb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d5c57bd97aa94d129a7ab85327649b57, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e7b631ecdd9741fea8d3da9c6189b635, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4c4dcf72ebdb4612a327847419ae6f2b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b35b8ef8de944c688e17c598905826fc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ed368f9261634c4dbd1ab1c48a9390a8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/63b602174d5048c9a5109fe68b02017a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/da982e863e5844da9b2d99c279f8c03c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b2f4e1e345a49da8fbc2cd4e0faefd4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4f3fa1d37de04c58a9b5ec5974484cdc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b7a5ae7b9cc48dca76bbc15cef6c0cf, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/f32df6ef3faf43028586219ffba20800, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/74b8eb2c28224032802c2d6a5eb87f66, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/0dcceecf38694396b32fb5192370f8ba, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ebad7e17c5534d29b5e5865f9076086a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/6c0dc19f0de247ebb0c2f2f75f695543] to archive 2023-04-16 20:56:26,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:56:26,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2704a0642d9a46e9a8c213f04e042334 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2704a0642d9a46e9a8c213f04e042334 2023-04-16 20:56:26,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e502e329789a42219dd2453516f8edf3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e502e329789a42219dd2453516f8edf3 2023-04-16 20:56:26,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/32900ffa448042b096c346e1c8a87a79 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/32900ffa448042b096c346e1c8a87a79 2023-04-16 20:56:26,177 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2675171c9ed7406e8fb4184926925ac2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2675171c9ed7406e8fb4184926925ac2 2023-04-16 20:56:26,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3aa158af4c21488e9c6af541fa26f2b6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3aa158af4c21488e9c6af541fa26f2b6 2023-04-16 20:56:26,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ec3bba95fab74452a9e1249d94deeff0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ec3bba95fab74452a9e1249d94deeff0 2023-04-16 20:56:26,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/9d9e036edd414faf93955a3bffb5d23a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/9d9e036edd414faf93955a3bffb5d23a 2023-04-16 20:56:26,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e6abc28f59344518bf95b9ec6f27066b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e6abc28f59344518bf95b9ec6f27066b 2023-04-16 20:56:26,181 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8622eb9a357948d6ac0b3d14b7ee0bff to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8622eb9a357948d6ac0b3d14b7ee0bff 2023-04-16 20:56:26,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/102a79efb4634694a7fe07d970d74df5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/102a79efb4634694a7fe07d970d74df5 2023-04-16 20:56:26,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c9a95d1ea7bc49f4a9e57d6666408314 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c9a95d1ea7bc49f4a9e57d6666408314 2023-04-16 20:56:26,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c2e53985332c4d2f90055e53ca30407f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c2e53985332c4d2f90055e53ca30407f 2023-04-16 20:56:26,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ddb9ccf568564db2abbd4943da7c8324 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ddb9ccf568564db2abbd4943da7c8324 2023-04-16 20:56:26,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b630a135cfe44963b749c0ad452a2280 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b630a135cfe44963b749c0ad452a2280 2023-04-16 20:56:26,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/a65af645b2d54fcf83027ebfed929062 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/a65af645b2d54fcf83027ebfed929062 2023-04-16 20:56:26,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/86ee031a103745c38fa6d7471a639bd9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/86ee031a103745c38fa6d7471a639bd9 2023-04-16 20:56:26,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c96838fe89eb40d98abdf27cf3aa4193 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c96838fe89eb40d98abdf27cf3aa4193 2023-04-16 20:56:26,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3f511669c2af4ea0b3f5c03dcfb7640b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3f511669c2af4ea0b3f5c03dcfb7640b 2023-04-16 20:56:26,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c21b67d588d0416cb8e422fe13d8889e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/c21b67d588d0416cb8e422fe13d8889e 2023-04-16 20:56:26,189 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/95da552ee04e415b9604801ca84d9ed6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/95da552ee04e415b9604801ca84d9ed6 2023-04-16 20:56:26,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7ca748abe6064bf7af8f48a688151b71 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7ca748abe6064bf7af8f48a688151b71 2023-04-16 20:56:26,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/5b6a5a3dcda547d6b56feee69072111e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/5b6a5a3dcda547d6b56feee69072111e 2023-04-16 20:56:26,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2b92c60bd0494472b257320163258119 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/2b92c60bd0494472b257320163258119 2023-04-16 20:56:26,192 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/60537b2702924acbacccf9eafd6c5e63 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/60537b2702924acbacccf9eafd6c5e63 2023-04-16 20:56:26,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/eb8013ea1e5a40bd8b04158ddd59af2e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/eb8013ea1e5a40bd8b04158ddd59af2e 2023-04-16 20:56:26,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3fd29fca8db04faab62650a2f2269d05 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/3fd29fca8db04faab62650a2f2269d05 2023-04-16 20:56:26,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7eaecce4085e4fb9a0ca5e11e0e8401d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7eaecce4085e4fb9a0ca5e11e0e8401d 2023-04-16 20:56:26,195 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7a9fb73a6ef8400eb91907db51f3feb7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7a9fb73a6ef8400eb91907db51f3feb7 2023-04-16 20:56:26,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/191a56633f6b4b3d854728ba90062b90 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/191a56633f6b4b3d854728ba90062b90 2023-04-16 20:56:26,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d9216693d2274e4abf2aaa2dc340e340 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d9216693d2274e4abf2aaa2dc340e340 2023-04-16 20:56:26,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/132d5ec25f5d4081bea78753e18ff2a8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/132d5ec25f5d4081bea78753e18ff2a8 2023-04-16 20:56:26,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8313c971958a4a5e9e3d144ea6ea36eb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8313c971958a4a5e9e3d144ea6ea36eb 2023-04-16 20:56:26,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d5c57bd97aa94d129a7ab85327649b57 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/d5c57bd97aa94d129a7ab85327649b57 2023-04-16 20:56:26,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e7b631ecdd9741fea8d3da9c6189b635 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/e7b631ecdd9741fea8d3da9c6189b635 2023-04-16 20:56:26,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4c4dcf72ebdb4612a327847419ae6f2b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4c4dcf72ebdb4612a327847419ae6f2b 2023-04-16 20:56:26,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b35b8ef8de944c688e17c598905826fc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/b35b8ef8de944c688e17c598905826fc 2023-04-16 20:56:26,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ed368f9261634c4dbd1ab1c48a9390a8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ed368f9261634c4dbd1ab1c48a9390a8 2023-04-16 20:56:26,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/63b602174d5048c9a5109fe68b02017a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/63b602174d5048c9a5109fe68b02017a 2023-04-16 20:56:26,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/da982e863e5844da9b2d99c279f8c03c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/da982e863e5844da9b2d99c279f8c03c 2023-04-16 20:56:26,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b2f4e1e345a49da8fbc2cd4e0faefd4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b2f4e1e345a49da8fbc2cd4e0faefd4 2023-04-16 20:56:26,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4f3fa1d37de04c58a9b5ec5974484cdc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4f3fa1d37de04c58a9b5ec5974484cdc 2023-04-16 20:56:26,205 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b7a5ae7b9cc48dca76bbc15cef6c0cf to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/8b7a5ae7b9cc48dca76bbc15cef6c0cf 2023-04-16 20:56:26,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/f32df6ef3faf43028586219ffba20800 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/f32df6ef3faf43028586219ffba20800 2023-04-16 20:56:26,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/74b8eb2c28224032802c2d6a5eb87f66 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/74b8eb2c28224032802c2d6a5eb87f66 2023-04-16 20:56:26,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/0dcceecf38694396b32fb5192370f8ba to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/0dcceecf38694396b32fb5192370f8ba 2023-04-16 20:56:26,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ebad7e17c5534d29b5e5865f9076086a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/ebad7e17c5534d29b5e5865f9076086a 2023-04-16 20:56:26,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/6c0dc19f0de247ebb0c2f2f75f695543 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/6c0dc19f0de247ebb0c2f2f75f695543 2023-04-16 20:56:26,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb9b03a0022e455692b1d3ad78331d43, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/65eb332deac942bfafb96a2ea62a26f3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1f527b07382e410bb8ba2f290ed063fc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0068cd5599604468a173dc0e23a79290, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0819f206a4a64dfd95401f73ef8e575a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1db593803dce49a28a57aad6e6892118, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5d5e53bf582f4c42ac3b99fc4402cd80, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/62963c595aef48e7b1ff4bc1554f6221, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/34d8103930b34ab49f50d2046a7cad64, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/30283da273894fea835a386d5c3976e1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/175766a40d6044fdbdfb8b9776543013, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1310bd0c441047248f10b140e85c8465, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/6dbf9453875943c58b7545c200690325, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f6d75490a3a44a14b3e295bb73aad011, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/9b5572e2907f49e39a2907891d808b18, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b6756ca5e4764e938e2c3f6a9fe46ffd, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d97a4c03611d4e7fadd102bcd7c9e532, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/70d8851e511d40e4be5dcc77f1f9a8c4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/434850c28ff046d4b2c025560ab8261f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0c0e06f0d1694a4ba2c8ee56dcc19d71, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ade1a505568048e4bc14944b1c11f6a0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/43ed3dc847ef440b8369dbc1aaa819fb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b67b0936a8cf49f5b34fd2a9eb5c00e1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5bd8c2bb01c141548250b80f34ec9420, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/664a1727d73440079db194b9c0ae50a4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a8d0213f7bf44fafab184524477842d6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2da2bd364f3a4ef391255f6df272e3b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/05c4f82ef2ab41ee8fc7bbac88468295, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/e6c62aeec51b4c29b207209c75936eb9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ba99d2c7a5c749c1b844572a12bcf3ba, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2ef29875a14844ea8842dede2240c123, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3d25378aefb14d8fb8207839cb7cbb45, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3fb94b5fe6ff47a1baa1afbb7fbd1e7e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f8d471d7c9794fb8994f23033fc60c2e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb17dde083c64e3f95c216bbe4894d40, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8fd66e14399e42ec95505e5596d8a7f6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0ad3031e9fc548a6af9e251e093edeb0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a32216deba7246ceb62a9fddbb9bbf55, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/83fa794c6dcb4f27a4c8c05437a29fc7, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/11a6b56d13054fa6b9ee5edce68c5abc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8aa612e834f54dcea0429f3365cff1e4, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d23f01f5dfb04015a5ef730baed3dc61, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/917b4b8e33154e75a36268c3c3240c28, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/948e95a0873e4f0fbcef134dd7c4b66b, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1ba427eee644a319fb787b44504c9ca, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/be8eb4dadb48441992556b5e8b0b06cb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/55a072de59fd43f7ad5b1551a68eae89] to archive 2023-04-16 20:56:26,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:56:26,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb9b03a0022e455692b1d3ad78331d43 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb9b03a0022e455692b1d3ad78331d43 2023-04-16 20:56:26,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/65eb332deac942bfafb96a2ea62a26f3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/65eb332deac942bfafb96a2ea62a26f3 2023-04-16 20:56:26,217 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1f527b07382e410bb8ba2f290ed063fc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1f527b07382e410bb8ba2f290ed063fc 2023-04-16 20:56:26,218 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0068cd5599604468a173dc0e23a79290 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0068cd5599604468a173dc0e23a79290 2023-04-16 20:56:26,218 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0819f206a4a64dfd95401f73ef8e575a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0819f206a4a64dfd95401f73ef8e575a 2023-04-16 20:56:26,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1db593803dce49a28a57aad6e6892118 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1db593803dce49a28a57aad6e6892118 2023-04-16 20:56:26,220 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5d5e53bf582f4c42ac3b99fc4402cd80 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5d5e53bf582f4c42ac3b99fc4402cd80 2023-04-16 20:56:26,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/62963c595aef48e7b1ff4bc1554f6221 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/62963c595aef48e7b1ff4bc1554f6221 2023-04-16 20:56:26,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/34d8103930b34ab49f50d2046a7cad64 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/34d8103930b34ab49f50d2046a7cad64 2023-04-16 20:56:26,222 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/30283da273894fea835a386d5c3976e1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/30283da273894fea835a386d5c3976e1 2023-04-16 20:56:26,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/175766a40d6044fdbdfb8b9776543013 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/175766a40d6044fdbdfb8b9776543013 2023-04-16 20:56:26,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1310bd0c441047248f10b140e85c8465 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/1310bd0c441047248f10b140e85c8465 2023-04-16 20:56:26,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/6dbf9453875943c58b7545c200690325 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/6dbf9453875943c58b7545c200690325 2023-04-16 20:56:26,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f6d75490a3a44a14b3e295bb73aad011 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f6d75490a3a44a14b3e295bb73aad011 2023-04-16 20:56:26,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/9b5572e2907f49e39a2907891d808b18 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/9b5572e2907f49e39a2907891d808b18 2023-04-16 20:56:26,226 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b6756ca5e4764e938e2c3f6a9fe46ffd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b6756ca5e4764e938e2c3f6a9fe46ffd 2023-04-16 20:56:26,227 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d97a4c03611d4e7fadd102bcd7c9e532 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d97a4c03611d4e7fadd102bcd7c9e532 2023-04-16 20:56:26,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/70d8851e511d40e4be5dcc77f1f9a8c4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/70d8851e511d40e4be5dcc77f1f9a8c4 2023-04-16 20:56:26,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/434850c28ff046d4b2c025560ab8261f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/434850c28ff046d4b2c025560ab8261f 2023-04-16 20:56:26,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0c0e06f0d1694a4ba2c8ee56dcc19d71 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0c0e06f0d1694a4ba2c8ee56dcc19d71 2023-04-16 20:56:26,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ade1a505568048e4bc14944b1c11f6a0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ade1a505568048e4bc14944b1c11f6a0 2023-04-16 20:56:26,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/43ed3dc847ef440b8369dbc1aaa819fb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/43ed3dc847ef440b8369dbc1aaa819fb 2023-04-16 20:56:26,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b67b0936a8cf49f5b34fd2a9eb5c00e1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/b67b0936a8cf49f5b34fd2a9eb5c00e1 2023-04-16 20:56:26,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5bd8c2bb01c141548250b80f34ec9420 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/5bd8c2bb01c141548250b80f34ec9420 2023-04-16 20:56:26,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/664a1727d73440079db194b9c0ae50a4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/664a1727d73440079db194b9c0ae50a4 2023-04-16 20:56:26,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a8d0213f7bf44fafab184524477842d6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a8d0213f7bf44fafab184524477842d6 2023-04-16 20:56:26,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2da2bd364f3a4ef391255f6df272e3b0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2da2bd364f3a4ef391255f6df272e3b0 2023-04-16 20:56:26,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/05c4f82ef2ab41ee8fc7bbac88468295 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/05c4f82ef2ab41ee8fc7bbac88468295 2023-04-16 20:56:26,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/e6c62aeec51b4c29b207209c75936eb9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/e6c62aeec51b4c29b207209c75936eb9 2023-04-16 20:56:26,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ba99d2c7a5c749c1b844572a12bcf3ba to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/ba99d2c7a5c749c1b844572a12bcf3ba 2023-04-16 20:56:26,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2ef29875a14844ea8842dede2240c123 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/2ef29875a14844ea8842dede2240c123 2023-04-16 20:56:26,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3d25378aefb14d8fb8207839cb7cbb45 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3d25378aefb14d8fb8207839cb7cbb45 2023-04-16 20:56:26,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3fb94b5fe6ff47a1baa1afbb7fbd1e7e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/3fb94b5fe6ff47a1baa1afbb7fbd1e7e 2023-04-16 20:56:26,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f8d471d7c9794fb8994f23033fc60c2e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/f8d471d7c9794fb8994f23033fc60c2e 2023-04-16 20:56:26,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb17dde083c64e3f95c216bbe4894d40 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/bb17dde083c64e3f95c216bbe4894d40 2023-04-16 20:56:26,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8fd66e14399e42ec95505e5596d8a7f6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8fd66e14399e42ec95505e5596d8a7f6 2023-04-16 20:56:26,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0ad3031e9fc548a6af9e251e093edeb0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/0ad3031e9fc548a6af9e251e093edeb0 2023-04-16 20:56:26,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a32216deba7246ceb62a9fddbb9bbf55 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/a32216deba7246ceb62a9fddbb9bbf55 2023-04-16 20:56:26,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/83fa794c6dcb4f27a4c8c05437a29fc7 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/83fa794c6dcb4f27a4c8c05437a29fc7 2023-04-16 20:56:26,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/11a6b56d13054fa6b9ee5edce68c5abc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/11a6b56d13054fa6b9ee5edce68c5abc 2023-04-16 20:56:26,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8aa612e834f54dcea0429f3365cff1e4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/8aa612e834f54dcea0429f3365cff1e4 2023-04-16 20:56:26,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d23f01f5dfb04015a5ef730baed3dc61 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d23f01f5dfb04015a5ef730baed3dc61 2023-04-16 20:56:26,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/917b4b8e33154e75a36268c3c3240c28 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/917b4b8e33154e75a36268c3c3240c28 2023-04-16 20:56:26,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/948e95a0873e4f0fbcef134dd7c4b66b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/948e95a0873e4f0fbcef134dd7c4b66b 2023-04-16 20:56:26,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1ba427eee644a319fb787b44504c9ca to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1ba427eee644a319fb787b44504c9ca 2023-04-16 20:56:26,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/be8eb4dadb48441992556b5e8b0b06cb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/be8eb4dadb48441992556b5e8b0b06cb 2023-04-16 20:56:26,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/55a072de59fd43f7ad5b1551a68eae89 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/55a072de59fd43f7ad5b1551a68eae89 2023-04-16 20:56:26,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/94f436f7162c49a5a17d38d413a49cef, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f54324efdf8b4c6b9fb9bb0148380138, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/47a95e71c90e4590aa6ef549a2f055ee, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/994f25c3f5364a64803c422daec6f35c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/501a94e96ee341e8b259f4098faf7156, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/1ebb97974a034fe390da0f76071a4d4c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f9c588a5dae04dfc8e144593068382a2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8facf058f6e447b689a75e6fb32c5287, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2546a18582bb4ac79e574f828b9263fb, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ca561e90749d46be9eb9648f089c9319, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cd5ecbc5c4f74c3e9f56b552e18b28a9, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ceebbbec6a95483cb8642f6d066e7ce3, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ac25876d45684ca5b7d462a7310d62d0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e94da69735344add89381974d11eba68, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/952f6265721049ffa8b5f7affdc85e5f, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/536904cc5a684e12b23512fd9ced6cae, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d22bcb8c03484fa48db84e497127d443, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e3e27a0af1734605b96323327532eec2, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/51003de5977e4465ad1330b95d15070c, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/10917cc419b34a3d9d1c108152f20908, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cf901dca34d8436590a5f6db99ed3302, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/206e14bf06a44445864681e3736b3070, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/917d9c60cf664c7cb835db770b54b46d, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e896d5cfbd4045bcb4eb91b96e62506a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8321063e190d4f40b96aed04d922ba92, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/b6154ba414f9446d8adb39b9f38c6db6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2772978fc3ec444e9fe49c45a2a2ccde, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/58ebc6aebdca46a48998eed253cca1f1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c52e42c3057446dcb01d851e724d6fa6, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/6cd98d43825947d9892998fa41198e34, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/bffe3b5ff60b400b8f7cbc340a778596, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/600583335cfe4dba983a2b601e6654dc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/7b6208ef6de0457d9020a09e7e41ecf1, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/fbf25c4f9b974921adf51f46b76f47bc, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/05a4428badd84509a620e249702aaeee, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/69392604f5dd4710b0ea8a9bbc4294b8, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/15413fa2f2ff4e6aa155b1853705ea6a, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ea797382212e465cbdc8cf298565e032, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/a90094462daa4b08836f5412b0eb074e, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/de8af74a56ab45b09b21df43690bc0b0, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cb98152b721244e08200dac603abd321, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/db783452d5b94df28429c4cdf2c74a21, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/5196bff1881441edbd7d03338ff7aa92, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c71d5f09bbe540daa925e3ebe5da4c46, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/668982571b23452eaefe18bf714578ba] to archive 2023-04-16 20:56:26,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 20:56:26,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/94f436f7162c49a5a17d38d413a49cef to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/94f436f7162c49a5a17d38d413a49cef 2023-04-16 20:56:26,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f54324efdf8b4c6b9fb9bb0148380138 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f54324efdf8b4c6b9fb9bb0148380138 2023-04-16 20:56:26,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/47a95e71c90e4590aa6ef549a2f055ee to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/47a95e71c90e4590aa6ef549a2f055ee 2023-04-16 20:56:26,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/994f25c3f5364a64803c422daec6f35c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/994f25c3f5364a64803c422daec6f35c 2023-04-16 20:56:26,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/501a94e96ee341e8b259f4098faf7156 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/501a94e96ee341e8b259f4098faf7156 2023-04-16 20:56:26,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/1ebb97974a034fe390da0f76071a4d4c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/1ebb97974a034fe390da0f76071a4d4c 2023-04-16 20:56:26,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f9c588a5dae04dfc8e144593068382a2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/f9c588a5dae04dfc8e144593068382a2 2023-04-16 20:56:26,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8facf058f6e447b689a75e6fb32c5287 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8facf058f6e447b689a75e6fb32c5287 2023-04-16 20:56:26,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2546a18582bb4ac79e574f828b9263fb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2546a18582bb4ac79e574f828b9263fb 2023-04-16 20:56:26,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ca561e90749d46be9eb9648f089c9319 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ca561e90749d46be9eb9648f089c9319 2023-04-16 20:56:26,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cd5ecbc5c4f74c3e9f56b552e18b28a9 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cd5ecbc5c4f74c3e9f56b552e18b28a9 2023-04-16 20:56:26,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ceebbbec6a95483cb8642f6d066e7ce3 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ceebbbec6a95483cb8642f6d066e7ce3 2023-04-16 20:56:26,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ac25876d45684ca5b7d462a7310d62d0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ac25876d45684ca5b7d462a7310d62d0 2023-04-16 20:56:26,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e94da69735344add89381974d11eba68 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e94da69735344add89381974d11eba68 2023-04-16 20:56:26,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/952f6265721049ffa8b5f7affdc85e5f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/952f6265721049ffa8b5f7affdc85e5f 2023-04-16 20:56:26,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/536904cc5a684e12b23512fd9ced6cae to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/536904cc5a684e12b23512fd9ced6cae 2023-04-16 20:56:26,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d22bcb8c03484fa48db84e497127d443 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d22bcb8c03484fa48db84e497127d443 2023-04-16 20:56:26,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e3e27a0af1734605b96323327532eec2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e3e27a0af1734605b96323327532eec2 2023-04-16 20:56:26,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/51003de5977e4465ad1330b95d15070c to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/51003de5977e4465ad1330b95d15070c 2023-04-16 20:56:26,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/10917cc419b34a3d9d1c108152f20908 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/10917cc419b34a3d9d1c108152f20908 2023-04-16 20:56:26,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cf901dca34d8436590a5f6db99ed3302 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cf901dca34d8436590a5f6db99ed3302 2023-04-16 20:56:26,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/206e14bf06a44445864681e3736b3070 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/206e14bf06a44445864681e3736b3070 2023-04-16 20:56:26,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/917d9c60cf664c7cb835db770b54b46d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/917d9c60cf664c7cb835db770b54b46d 2023-04-16 20:56:26,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e896d5cfbd4045bcb4eb91b96e62506a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/e896d5cfbd4045bcb4eb91b96e62506a 2023-04-16 20:56:26,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8321063e190d4f40b96aed04d922ba92 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/8321063e190d4f40b96aed04d922ba92 2023-04-16 20:56:26,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/b6154ba414f9446d8adb39b9f38c6db6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/b6154ba414f9446d8adb39b9f38c6db6 2023-04-16 20:56:26,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2772978fc3ec444e9fe49c45a2a2ccde to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/2772978fc3ec444e9fe49c45a2a2ccde 2023-04-16 20:56:26,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/58ebc6aebdca46a48998eed253cca1f1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/58ebc6aebdca46a48998eed253cca1f1 2023-04-16 20:56:26,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c52e42c3057446dcb01d851e724d6fa6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c52e42c3057446dcb01d851e724d6fa6 2023-04-16 20:56:26,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/6cd98d43825947d9892998fa41198e34 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/6cd98d43825947d9892998fa41198e34 2023-04-16 20:56:26,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/bffe3b5ff60b400b8f7cbc340a778596 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/bffe3b5ff60b400b8f7cbc340a778596 2023-04-16 20:56:26,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/600583335cfe4dba983a2b601e6654dc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/600583335cfe4dba983a2b601e6654dc 2023-04-16 20:56:26,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/7b6208ef6de0457d9020a09e7e41ecf1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/7b6208ef6de0457d9020a09e7e41ecf1 2023-04-16 20:56:26,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/fbf25c4f9b974921adf51f46b76f47bc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/fbf25c4f9b974921adf51f46b76f47bc 2023-04-16 20:56:26,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/05a4428badd84509a620e249702aaeee to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/05a4428badd84509a620e249702aaeee 2023-04-16 20:56:26,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/69392604f5dd4710b0ea8a9bbc4294b8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/69392604f5dd4710b0ea8a9bbc4294b8 2023-04-16 20:56:26,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/15413fa2f2ff4e6aa155b1853705ea6a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/15413fa2f2ff4e6aa155b1853705ea6a 2023-04-16 20:56:26,287 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ea797382212e465cbdc8cf298565e032 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/ea797382212e465cbdc8cf298565e032 2023-04-16 20:56:26,288 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/a90094462daa4b08836f5412b0eb074e to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/a90094462daa4b08836f5412b0eb074e 2023-04-16 20:56:26,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/de8af74a56ab45b09b21df43690bc0b0 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/de8af74a56ab45b09b21df43690bc0b0 2023-04-16 20:56:26,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cb98152b721244e08200dac603abd321 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/cb98152b721244e08200dac603abd321 2023-04-16 20:56:26,290 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/db783452d5b94df28429c4cdf2c74a21 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/db783452d5b94df28429c4cdf2c74a21 2023-04-16 20:56:26,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/5196bff1881441edbd7d03338ff7aa92 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/5196bff1881441edbd7d03338ff7aa92 2023-04-16 20:56:26,292 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c71d5f09bbe540daa925e3ebe5da4c46 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/c71d5f09bbe540daa925e3ebe5da4c46 2023-04-16 20:56:26,292 DEBUG [StoreCloser-TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/668982571b23452eaefe18bf714578ba to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/668982571b23452eaefe18bf714578ba 2023-04-16 20:56:26,295 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/recovered.edits/850.seqid, newMaxSeqId=850, maxSeqId=4 2023-04-16 20:56:26,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40. 2023-04-16 20:56:26,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9853af3382d774c4b02dae6a5c604d40: 2023-04-16 20:56:26,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:26,298 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=9853af3382d774c4b02dae6a5c604d40, regionState=CLOSED 2023-04-16 20:56:26,298 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681678586298"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681678586298"}]},"ts":"1681678586298"} 2023-04-16 20:56:26,300 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-16 20:56:26,300 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 9853af3382d774c4b02dae6a5c604d40, server=jenkins-hbase4.apache.org,41211,1681678416093 in 1.5200 sec 2023-04-16 20:56:26,301 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-16 20:56:26,301 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9853af3382d774c4b02dae6a5c604d40, UNASSIGN in 1.5240 sec 2023-04-16 20:56:26,301 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681678586301"}]},"ts":"1681678586301"} 2023-04-16 20:56:26,302 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 20:56:26,304 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 20:56:26,305 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5340 sec 2023-04-16 20:56:26,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 20:56:26,878 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-16 20:56:26,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 20:56:26,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:56:26,881 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:56:26,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-16 20:56:26,881 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:56:26,885 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:26,887 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C, FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/recovered.edits] 2023-04-16 20:56:26,889 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4b45eefc0fd849bfaec0809b81785e1a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/4b45eefc0fd849bfaec0809b81785e1a 2023-04-16 20:56:26,890 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7f553d2e50d449bb9be97f75f9c04e1d to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/A/7f553d2e50d449bb9be97f75f9c04e1d 2023-04-16 20:56:26,892 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1cdacd3ecf2439f89eb3b5ee67eb79a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d1cdacd3ecf2439f89eb3b5ee67eb79a 2023-04-16 20:56:26,893 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d269d20521b34f1289f00ea6bcbfe7a4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/B/d269d20521b34f1289f00ea6bcbfe7a4 2023-04-16 20:56:26,895 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d19522945082449eac4a165f988e23cc to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/d19522945082449eac4a165f988e23cc 2023-04-16 20:56:26,896 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/fcf51076f6314d9c91aed93579243359 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/C/fcf51076f6314d9c91aed93579243359 2023-04-16 20:56:26,898 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/recovered.edits/850.seqid to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40/recovered.edits/850.seqid 2023-04-16 20:56:26,898 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/.tmp/data/default/TestAcidGuarantees/9853af3382d774c4b02dae6a5c604d40 2023-04-16 20:56:26,898 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 20:56:26,899 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 20:56:26,899 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-16 20:56:26,902 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041602157a54b51b4b718c6bc5f0b664736a to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041602157a54b51b4b718c6bc5f0b664736a 2023-04-16 20:56:26,903 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416096907b68e55455cabbca11c7ef1dd50 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416096907b68e55455cabbca11c7ef1dd50 2023-04-16 20:56:26,904 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041612545c76b9fa4835ba1cd68d11748738 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041612545c76b9fa4835ba1cd68d11748738 2023-04-16 20:56:26,905 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041618a2e6912c7a49df989ec6860936c962 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041618a2e6912c7a49df989ec6860936c962 2023-04-16 20:56:26,905 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416199e3e79d1d541529fe12953ecbf5629 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416199e3e79d1d541529fe12953ecbf5629 2023-04-16 20:56:26,906 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161a8767e04aba417283c94cadadf72a3f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161a8767e04aba417283c94cadadf72a3f 2023-04-16 20:56:26,907 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161b5be5d1557f4d8fb57299311b10dbe8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161b5be5d1557f4d8fb57299311b10dbe8 2023-04-16 20:56:26,908 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041621c7473c035c4ad1ab177ac800b54e1b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041621c7473c035c4ad1ab177ac800b54e1b 2023-04-16 20:56:26,909 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041625e102713ba64bfaa2d65830f32e80f8 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041625e102713ba64bfaa2d65830f32e80f8 2023-04-16 20:56:26,910 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416264aec2b595348f6b539555a23684beb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416264aec2b595348f6b539555a23684beb 2023-04-16 20:56:26,911 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163550395fa2ce458093693e6f96275ad1 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163550395fa2ce458093693e6f96275ad1 2023-04-16 20:56:26,912 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163b92e7a295124d639dfa2dd219bc3621 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163b92e7a295124d639dfa2dd219bc3621 2023-04-16 20:56:26,913 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164618b0dc99d04def9a3cf05b12aaaf69 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164618b0dc99d04def9a3cf05b12aaaf69 2023-04-16 20:56:26,914 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416554d6af26b4746eb9df50a44c3611073 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416554d6af26b4746eb9df50a44c3611073 2023-04-16 20:56:26,915 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416560324370660428f884c98b1c7a12def to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416560324370660428f884c98b1c7a12def 2023-04-16 20:56:26,915 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416576fa48476cd40d08dc98e6a8f220dcd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416576fa48476cd40d08dc98e6a8f220dcd 2023-04-16 20:56:26,916 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166207d93ff46d45a1b872b82bdc28b1f2 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166207d93ff46d45a1b872b82bdc28b1f2 2023-04-16 20:56:26,917 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416639c54d8e045410a8db055a53039d14f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416639c54d8e045410a8db055a53039d14f 2023-04-16 20:56:26,918 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416659643ccd8da4fe3ad8e5061f12cdddd to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416659643ccd8da4fe3ad8e5061f12cdddd 2023-04-16 20:56:26,919 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166ac830aa52c7491b9cc103600d0e5096 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166ac830aa52c7491b9cc103600d0e5096 2023-04-16 20:56:26,920 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166bdc5a78baa5460098a442a383785bd5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166bdc5a78baa5460098a442a383785bd5 2023-04-16 20:56:26,921 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167b0722c036714ac08b782162a393ec86 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167b0722c036714ac08b782162a393ec86 2023-04-16 20:56:26,921 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167c1c6d3d786642959d643f170f59f5e6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167c1c6d3d786642959d643f170f59f5e6 2023-04-16 20:56:26,922 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416872cd7a2fd114be68bd877c428f0bc59 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416872cd7a2fd114be68bd877c428f0bc59 2023-04-16 20:56:26,923 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169279198d680746c09d5d422865ebafe6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169279198d680746c09d5d422865ebafe6 2023-04-16 20:56:26,924 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041693d95b269f07424abea9bb53ebe60298 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041693d95b269f07424abea9bb53ebe60298 2023-04-16 20:56:26,925 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169a861cfc58564cfc8e16b2a8edd494d4 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169a861cfc58564cfc8e16b2a8edd494d4 2023-04-16 20:56:26,926 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416acfff2d5e5f147a6b00339b31a4bb4c5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416acfff2d5e5f147a6b00339b31a4bb4c5 2023-04-16 20:56:26,926 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416aefba12179934ab2b49c33fce3c35cc5 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416aefba12179934ab2b49c33fce3c35cc5 2023-04-16 20:56:26,927 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416af7b542f67b44c3cb93cb91a2df4d4bb to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416af7b542f67b44c3cb93cb91a2df4d4bb 2023-04-16 20:56:26,928 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b98e0f4c1e7e4c8ebc462be10a8119c6 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b98e0f4c1e7e4c8ebc462be10a8119c6 2023-04-16 20:56:26,929 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bd8cdf8f90e143afad2799ab7435318f to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bd8cdf8f90e143afad2799ab7435318f 2023-04-16 20:56:26,930 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c5f3dc5fa0394a61921b79a2ab23787b to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c5f3dc5fa0394a61921b79a2ab23787b 2023-04-16 20:56:26,931 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e28ad5a0f4eb4fb68754685687621c37 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e28ad5a0f4eb4fb68754685687621c37 2023-04-16 20:56:26,932 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e725a84772ba4e28bc01a660d222ecbe to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e725a84772ba4e28bc01a660d222ecbe 2023-04-16 20:56:26,932 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416efce844fccce4b1da3007f92320cc534 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416efce844fccce4b1da3007f92320cc534 2023-04-16 20:56:26,933 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f8fdee967ab94e51a4080b811a781a83 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f8fdee967ab94e51a4080b811a781a83 2023-04-16 20:56:26,934 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416fae5671b1526459c9dfd12f39d582735 to hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416fae5671b1526459c9dfd12f39d582735 2023-04-16 20:56:26,934 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 20:56:26,936 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:56:26,937 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 20:56:26,938 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 20:56:26,939 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:56:26,939 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 20:56:26,939 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681678586939"}]},"ts":"9223372036854775807"} 2023-04-16 20:56:26,940 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 20:56:26,940 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9853af3382d774c4b02dae6a5c604d40, NAME => 'TestAcidGuarantees,,1681678553514.9853af3382d774c4b02dae6a5c604d40.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 20:56:26,940 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 20:56:26,940 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681678586940"}]},"ts":"9223372036854775807"} 2023-04-16 20:56:26,941 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 20:56:26,943 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 20:56:26,943 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-04-16 20:56:26,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46091] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-16 20:56:26,982 INFO [Listener at localhost/45179] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-16 20:56:26,993 INFO [Listener at localhost/45179] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 236), OpenFileDescriptor=546 (was 601), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=362 (was 428), ProcessCount=169 (was 169), AvailableMemoryMB=924 (was 823) - AvailableMemoryMB LEAK? - 2023-04-16 20:56:26,993 INFO [Listener at localhost/45179] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-16 20:56:26,994 INFO [Listener at localhost/45179] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 20:56:26,994 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:54578 2023-04-16 20:56:26,994 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:26,994 DEBUG [Listener at localhost/45179] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-16 20:56:26,994 DEBUG [Listener at localhost/45179] util.JVMClusterUtil(257): Found active master hash=62412031, stopped=false 2023-04-16 20:56:26,994 INFO [Listener at localhost/45179] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:56:26,995 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-16 20:56:26,996 INFO [Listener at localhost/45179] procedure2.ProcedureExecutor(629): Stopping 2023-04-16 20:56:26,996 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:56:26,995 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-16 20:56:26,996 DEBUG [Listener at localhost/45179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0020e3a3 to 127.0.0.1:54578 2023-04-16 20:56:26,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-16 20:56:26,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-16 20:56:26,996 DEBUG [Listener at localhost/45179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:26,997 INFO [Listener at localhost/45179] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41211,1681678416093' ***** 2023-04-16 20:56:26,997 INFO [Listener at localhost/45179] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-16 20:56:26,997 INFO [RS:0;jenkins-hbase4:41211] regionserver.HeapMemoryManager(220): Stopping 2023-04-16 20:56:26,997 INFO [RS:0;jenkins-hbase4:41211] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-16 20:56:26,997 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-16 20:56:26,997 INFO [RS:0;jenkins-hbase4:41211] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-16 20:56:26,997 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(3304): Received CLOSE for 707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:56:26,998 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:26,998 DEBUG [RS:0;jenkins-hbase4:41211] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7801a339 to 127.0.0.1:54578 2023-04-16 20:56:26,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 707804278a44b6c4590cd788e4bf28c6, disabling compactions & flushes 2023-04-16 20:56:26,999 DEBUG [RS:0;jenkins-hbase4:41211] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:26,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:56:26,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:56:26,999 INFO [RS:0;jenkins-hbase4:41211] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-16 20:56:26,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. after waiting 0 ms 2023-04-16 20:56:26,999 INFO [RS:0;jenkins-hbase4:41211] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-16 20:56:26,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:56:26,999 INFO [RS:0;jenkins-hbase4:41211] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-16 20:56:27,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 707804278a44b6c4590cd788e4bf28c6 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-16 20:56:27,000 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-16 20:56:27,000 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-16 20:56:27,000 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 707804278a44b6c4590cd788e4bf28c6=hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6.} 2023-04-16 20:56:27,000 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-16 20:56:27,000 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-16 20:56:27,000 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-16 20:56:27,000 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-16 20:56:27,000 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-16 20:56:27,001 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-16 20:56:27,002 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1505): Waiting on 1588230740, 707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:56:27,085 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-16 20:56:27,202 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1505): Waiting on 1588230740, 707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:56:27,402 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1505): Waiting on 1588230740, 707804278a44b6c4590cd788e4bf28c6 2023-04-16 20:56:27,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6/.tmp/info/998990663f9c42798bd3a7a986237415 2023-04-16 20:56:27,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6/.tmp/info/998990663f9c42798bd3a7a986237415 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6/info/998990663f9c42798bd3a7a986237415 2023-04-16 20:56:27,418 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/.tmp/info/13676bdf29bc455b90662f7446fd52fb 2023-04-16 20:56:27,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6/info/998990663f9c42798bd3a7a986237415, entries=2, sequenceid=6, filesize=4.8 K 2023-04-16 20:56:27,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 707804278a44b6c4590cd788e4bf28c6 in 421ms, sequenceid=6, compaction requested=false 2023-04-16 20:56:27,422 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 13676bdf29bc455b90662f7446fd52fb 2023-04-16 20:56:27,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/namespace/707804278a44b6c4590cd788e4bf28c6/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-16 20:56:27,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:56:27,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 707804278a44b6c4590cd788e4bf28c6: 2023-04-16 20:56:27,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681678418431.707804278a44b6c4590cd788e4bf28c6. 2023-04-16 20:56:27,603 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-16 20:56:27,803 DEBUG [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-16 20:56:27,828 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/.tmp/rep_barrier/4373437a4237480c816eea497e496f92 2023-04-16 20:56:27,831 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4373437a4237480c816eea497e496f92 2023-04-16 20:56:27,837 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/.tmp/table/cecf0e5a9f9747c3b6adb6f2e18e74e2 2023-04-16 20:56:27,839 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cecf0e5a9f9747c3b6adb6f2e18e74e2 2023-04-16 20:56:27,840 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/.tmp/info/13676bdf29bc455b90662f7446fd52fb as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/info/13676bdf29bc455b90662f7446fd52fb 2023-04-16 20:56:27,841 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-16 20:56:27,841 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-16 20:56:27,842 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 13676bdf29bc455b90662f7446fd52fb 2023-04-16 20:56:27,843 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/info/13676bdf29bc455b90662f7446fd52fb, entries=22, sequenceid=93, filesize=7.1 K 2023-04-16 20:56:27,844 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/.tmp/rep_barrier/4373437a4237480c816eea497e496f92 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/rep_barrier/4373437a4237480c816eea497e496f92 2023-04-16 20:56:27,846 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4373437a4237480c816eea497e496f92 2023-04-16 20:56:27,846 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/rep_barrier/4373437a4237480c816eea497e496f92, entries=6, sequenceid=93, filesize=5.5 K 2023-04-16 20:56:27,847 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/.tmp/table/cecf0e5a9f9747c3b6adb6f2e18e74e2 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/table/cecf0e5a9f9747c3b6adb6f2e18e74e2 2023-04-16 20:56:27,849 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cecf0e5a9f9747c3b6adb6f2e18e74e2 2023-04-16 20:56:27,849 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/table/cecf0e5a9f9747c3b6adb6f2e18e74e2, entries=9, sequenceid=93, filesize=5.5 K 2023-04-16 20:56:27,850 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 850ms, sequenceid=93, compaction requested=false 2023-04-16 20:56:27,854 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-16 20:56:27,854 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-16 20:56:27,856 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-16 20:56:27,856 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-16 20:56:27,856 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-16 20:56:28,003 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41211,1681678416093; all regions closed. 2023-04-16 20:56:28,011 DEBUG [RS:0;jenkins-hbase4:41211] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/oldWALs 2023-04-16 20:56:28,011 INFO [RS:0;jenkins-hbase4:41211] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41211%2C1681678416093.meta:.meta(num 1681678418169) 2023-04-16 20:56:28,015 DEBUG [RS:0;jenkins-hbase4:41211] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/oldWALs 2023-04-16 20:56:28,015 INFO [RS:0;jenkins-hbase4:41211] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41211%2C1681678416093:(num 1681678418005) 2023-04-16 20:56:28,015 DEBUG [RS:0;jenkins-hbase4:41211] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:28,015 INFO [RS:0;jenkins-hbase4:41211] regionserver.LeaseManager(133): Closed leases 2023-04-16 20:56:28,015 INFO [RS:0;jenkins-hbase4:41211] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-16 20:56:28,015 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-16 20:56:28,017 INFO [RS:0;jenkins-hbase4:41211] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41211 2023-04-16 20:56:28,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41211,1681678416093 2023-04-16 20:56:28,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-16 20:56:28,022 ERROR [Listener at localhost/45179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3059765d rejected from java.util.concurrent.ThreadPoolExecutor@3466e48e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2461] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 20:56:28,022 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-16 20:56:28,022 ERROR [Listener at localhost/45179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b4ffccb rejected from java.util.concurrent.ThreadPoolExecutor@3466e48e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2461] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 20:56:28,025 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41211,1681678416093] 2023-04-16 20:56:28,025 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41211,1681678416093; numProcessing=1 2023-04-16 20:56:28,026 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41211,1681678416093 already deleted, retry=false 2023-04-16 20:56:28,026 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,41211,1681678416093 expired; onlineServers=0 2023-04-16 20:56:28,026 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46091,1681678414874' ***** 2023-04-16 20:56:28,026 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-16 20:56:28,027 DEBUG [M:0;jenkins-hbase4:46091] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42855ec4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-16 20:56:28,027 INFO [M:0;jenkins-hbase4:46091] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46091,1681678414874 2023-04-16 20:56:28,027 INFO [M:0;jenkins-hbase4:46091] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46091,1681678414874; all regions closed. 2023-04-16 20:56:28,027 DEBUG [M:0;jenkins-hbase4:46091] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 20:56:28,027 DEBUG [M:0;jenkins-hbase4:46091] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-16 20:56:28,027 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-16 20:56:28,027 DEBUG [M:0;jenkins-hbase4:46091] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-16 20:56:28,027 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681678417660] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681678417660,5,FailOnTimeoutGroup] 2023-04-16 20:56:28,027 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681678417656] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681678417656,5,FailOnTimeoutGroup] 2023-04-16 20:56:28,027 INFO [M:0;jenkins-hbase4:46091] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-16 20:56:28,027 INFO [M:0;jenkins-hbase4:46091] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-16 20:56:28,027 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-16 20:56:28,028 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 20:56:28,028 INFO [M:0;jenkins-hbase4:46091] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-16 20:56:28,028 DEBUG [M:0;jenkins-hbase4:46091] master.HMaster(1512): Stopping service threads 2023-04-16 20:56:28,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-16 20:56:28,028 INFO [M:0;jenkins-hbase4:46091] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-16 20:56:28,029 ERROR [M:0;jenkins-hbase4:46091] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (1941167763) connection to localhost/127.0.0.1:38239 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-16 20:56:28,029 INFO [M:0;jenkins-hbase4:46091] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-16 20:56:28,029 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-16 20:56:28,030 DEBUG [M:0;jenkins-hbase4:46091] zookeeper.ZKUtil(398): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-16 20:56:28,030 WARN [M:0;jenkins-hbase4:46091] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-16 20:56:28,030 INFO [M:0;jenkins-hbase4:46091] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-16 20:56:28,030 INFO [M:0;jenkins-hbase4:46091] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-16 20:56:28,030 DEBUG [M:0;jenkins-hbase4:46091] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-16 20:56:28,030 INFO [M:0;jenkins-hbase4:46091] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 20:56:28,031 DEBUG [M:0;jenkins-hbase4:46091] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 20:56:28,031 DEBUG [M:0;jenkins-hbase4:46091] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-16 20:56:28,031 DEBUG [M:0;jenkins-hbase4:46091] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 20:56:28,031 INFO [M:0;jenkins-hbase4:46091] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.07 KB heapSize=393.75 KB 2023-04-16 20:56:28,039 INFO [M:0;jenkins-hbase4:46091] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.07 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c03a854c995c45ed9e912382b159d7c7 2023-04-16 20:56:28,042 INFO [M:0;jenkins-hbase4:46091] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c03a854c995c45ed9e912382b159d7c7 2023-04-16 20:56:28,043 DEBUG [M:0;jenkins-hbase4:46091] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c03a854c995c45ed9e912382b159d7c7 as hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c03a854c995c45ed9e912382b159d7c7 2023-04-16 20:56:28,045 INFO [M:0;jenkins-hbase4:46091] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c03a854c995c45ed9e912382b159d7c7 2023-04-16 20:56:28,045 INFO [M:0;jenkins-hbase4:46091] regionserver.HStore(1080): Added hdfs://localhost:38239/user/jenkins/test-data/7e2bccac-a098-1596-6acc-198bcdf121ad/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c03a854c995c45ed9e912382b159d7c7, entries=65, sequenceid=578, filesize=24.7 K 2023-04-16 20:56:28,046 INFO [M:0;jenkins-hbase4:46091] regionserver.HRegion(2948): Finished flush of dataSize ~345.07 KB/353355, heapSize ~393.73 KB/403184, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 15ms, sequenceid=578, compaction requested=false 2023-04-16 20:56:28,046 INFO [M:0;jenkins-hbase4:46091] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 20:56:28,046 DEBUG [M:0;jenkins-hbase4:46091] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-16 20:56:28,049 INFO [M:0;jenkins-hbase4:46091] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-16 20:56:28,049 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-16 20:56:28,050 INFO [M:0;jenkins-hbase4:46091] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46091 2023-04-16 20:56:28,051 DEBUG [M:0;jenkins-hbase4:46091] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46091,1681678414874 already deleted, retry=false 2023-04-16 20:56:28,126 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-16 20:56:28,126 INFO [RS:0;jenkins-hbase4:41211] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41211,1681678416093; zookeeper connection closed. 2023-04-16 20:56:28,126 ERROR [Listener at localhost/45179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@431e5060 rejected from java.util.concurrent.ThreadPoolExecutor@3466e48e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2461] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 20:56:28,126 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): regionserver:41211-0x1018ec01a880001, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-16 20:56:28,127 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@15c1155c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@15c1155c 2023-04-16 20:56:28,127 ERROR [Listener at localhost/45179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@55a68c6f rejected from java.util.concurrent.ThreadPoolExecutor@3466e48e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2461] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 20:56:28,127 INFO [Listener at localhost/45179] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-16 20:56:28,226 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-16 20:56:28,226 INFO [M:0;jenkins-hbase4:46091] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46091,1681678414874; zookeeper connection closed. 2023-04-16 20:56:28,226 ERROR [Listener at localhost/45179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@68f9d529 rejected from java.util.concurrent.ThreadPoolExecutor@30680d0b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4110] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 20:56:28,226 DEBUG [Listener at localhost/45179-EventThread] zookeeper.ZKWatcher(600): master:46091-0x1018ec01a880000, quorum=127.0.0.1:54578, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-16 20:56:28,226 ERROR [Listener at localhost/45179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7dea1dfb rejected from java.util.concurrent.ThreadPoolExecutor@30680d0b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4110] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 20:56:28,228 WARN [Listener at localhost/45179] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-16 20:56:28,232 INFO [Listener at localhost/45179] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-16 20:56:28,340 WARN [BP-1011464738-172.31.14.131-1681678411558 heartbeating to localhost/127.0.0.1:38239] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-16 20:56:28,340 WARN [BP-1011464738-172.31.14.131-1681678411558 heartbeating to localhost/127.0.0.1:38239] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1011464738-172.31.14.131-1681678411558 (Datanode Uuid e6ce2f5f-dc5a-45e0-9c0a-099fa881a9a6) service to localhost/127.0.0.1:38239 2023-04-16 20:56:28,343 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/cluster_f3674968-0d51-0b50-60ff-43cc95ffde93/dfs/data/data1/current/BP-1011464738-172.31.14.131-1681678411558] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-16 20:56:28,344 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/11141d08-60e5-eaf4-36b8-56db0400e7ac/cluster_f3674968-0d51-0b50-60ff-43cc95ffde93/dfs/data/data2/current/BP-1011464738-172.31.14.131-1681678411558] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-16 20:56:28,376 INFO [Listener at localhost/45179] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-16 20:56:28,562 INFO [Listener at localhost/45179] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-16 20:56:28,751 INFO [Listener at localhost/45179] hbase.HBaseTestingUtility(1293): Minicluster is down